Python 3 (ipykernel)
%load_ext autoreload%autoreload 2!pip install numpy torch wandb swig gymnasium[box2d] matplotlib termcolorzsh:1: no matches found: gymnasium[box2d]
import testfrom utils import *xxxxxxxxxx# Reinforcement Learning Part 1: DQNBy Lawrence Liu and Tonmoy Monsoor## Some General Instructions- As before, please keep the names of the layer consistent with what is requested in model.py. Otherwise the test functions will not work- You will need to fill in the model.py, the DQN.py file, the buffer.py file, and theenv_wrapper.pyDO NOT use Windows for this project, gymnasium does is not supported for windows and installing it will be a pain.By Lawrence Liu and Tonmoy Monsoor
As before, please keep the names of the layer consistent with what is requested in model.py. Otherwise the test functions will not work
You will need to fill in the model.py, the DQN.py file, the buffer.py file, and the env_wrapper.py
DO NOT use Windows for this project, gymnasium does is not supported for windows and installing it will be a pain.
xxxxxxxxxx### Introduction to the EnviromentWe will be training a DQN agent to play the game of CarRacing. The agent will be trained to play the game using the pixels of the game as an input. The reward structure is as follows for each frame:- -0.1 for each frame- +1000/N where N is the number of tiles visited by the car in the episodeThe overall goal of this game is to design a agent that is able to play the game with a average test score of above 600. In discrete mode the actions can take 5 actions,- 0: Do Nothing- 1: Turn Left- 2: Turn Right- 3: Accelerate- 4: BrakeFirst let us visualize the game and understand the environment.We will be training a DQN agent to play the game of CarRacing. The agent will be trained to play the game using the pixels of the game as an input. The reward structure is as follows for each frame:
The overall goal of this game is to design a agent that is able to play the game with a average test score of above 600. In discrete mode the actions can take 5 actions,
First let us visualize the game and understand the environment.
xxxxxxxxxximport gymnasium as gymimport numpy as npenv = gym.make('CarRacing-v2', continuous=False, render_mode='rgb_array')env.np_random = np.random.RandomState(42)xxxxxxxxxxfrom IPython.display import HTMLframes = []s, _ = env.reset()while True: a = env.action_space.sample() s, r, terminated, truncated, _ = env.step(a) frames.append(s) if terminated or truncated: breakanim = animate(frames)HTML(anim.to_jshtml())xxxxxxxxxxSo a couple things we can note:- at the beginning of the game, we have 50 frames of the game slowly zooming into the car, we should ignore this period, ie no-op during this period.- there is a black bar at the bottom of the screen, we should crop this out of the observation. In addition, another thing to note is that the current frame doesn't give much information about the velocity and acceleration of the car, and that the car does not move much for each frame.### Environment Wrapper (5 points)As a result, you will need to complete `EnvWrapper` in `env_wrapper.py`. You can find more information in the docstring for the wrapper, however the main idea is that it is a wrapper to the environment that does the following:- skips the first 50 frames of the game- crops out the black bar and reshapes the observation to a 84x84 image, as well as turning the resulting image to grayscale- performs the actions for `skip_frames` frames- stacks the last `num_frames` frames together to give the agent some information about the velocity and acceleration of the car.So a couple things we can note:
In addition, another thing to note is that the current frame doesn't give much information about the velocity and acceleration of the car, and that the car does not move much for each frame.
As a result, you will need to complete EnvWrapper in env_wrapper.py. You can find more information in the docstring for the wrapper, however the main idea is that it is a wrapper to the environment that does the following:
skip_frames framesnum_frames frames together to give the agent some information about the velocity and acceleration of the car.xxxxxxxxxxfrom env_wrapper import EnvWrappertest.test_wrapper(EnvWrapper)Passed reset Passed step
xxxxxxxxxx### CNN Model (5 points)Now we are ready to build the model. Our architecture of the CNN model is the one proposed by Mnih et al in "Human-level control through deep reinforcement learning". Specifically this consists of the following layers:- A convolutional layer with 32 filters of size 8x8 with stride 4 and relu activation- A convolutional layer with 64 filters of size 4x4 with stride 2 and relu activation- A convolutional layer with 64 filters of size 3x3 with stride 1 and relu activation- A fully connected layer with 512 units and relu activation- A fully connected layer with the number of outputs of the environmentPlease implement this model `Nature_Paper_Conv` in `model.py` as well as the helper `MLP` class.Now we are ready to build the model. Our architecture of the CNN model is the one proposed by Mnih et al in "Human-level control through deep reinforcement learning". Specifically this consists of the following layers:
Please implement this model Nature_Paper_Conv in model.py as well as the helper
MLP class.
xxxxxxxxxximport modeltest.test_model_DQN(model.Nature_Paper_Conv)Passed
xxxxxxxxxx### DQN (40 points)Now we are ready to implement the DQN algorithm. #### Replay Buffer (5 points)First start by implementing the DQN replay buffer `ReplayBufferDQN` in `buffer.py`. This buffer will store the transitions of the agent and sample them for training. xxxxxxxxxxfrom replay_buffer import ReplayBufferDQNtest.test_DQN_replay_buffer(ReplayBufferDQN)Passed
xxxxxxxxxx#### DQN (15 points)Now implement the `_optimize_model` and `sample_action` functions in `DQN` in `DQN.py`. The `_optimize_model` function will sample a batch of transitions from the replay buffer and update the model. The `sample_action` function will sample an action from the model given the current state. Train the model over 200 episdoes, validating every 50 episodes for 30 episodes, before testing the model for 50 episodes at the end.Now implement the _optimize_model and sample_action functions in DQN in DQN.py. The _optimize_model function will sample a batch of transitions from the replay buffer and update the model. The sample_action function will sample an action from the model given the current state. Train the model over 200 episdoes, validating every 50 episodes for 30 episodes, before testing the model for 50 episodes at the end.
xxxxxxxxxximport DQNimport utilsimport torch trainerDQN1 = DQN.DQN(EnvWrapper(env), model.Nature_Paper_Conv, lr = 0.00025, gamma = 0.95, buffer_size=100000, batch_size=32, loss_fn = "mse_loss", use_wandb = False, device = 'cpu', seed = 42, epsilon_scheduler = utils.exponential_decay(1, 700, 0.1), save_path = utils.get_save_path("DQN","./runs/"))out_mse = trainerDQN1.train(200,50,30,50,50) saving to ./runs/DQN/run0 Episode: 1: Time: 13.082035064697266 Total Reward: -57.71186440678014 Avg_Loss: 0.4895160892610527 Episode: 2: Time: 17.856481075286865 Total Reward: -48.736654804271 Avg_Loss: 0.5356716240419686 Episode: 3: Time: 17.907886743545532 Total Reward: -62.34693877551075 Avg_Loss: 0.6517468516581825 Episode: 4: Time: 18.092222929000854 Total Reward: -7.547528517110129 Avg_Loss: 0.5748330331994456 Episode: 5: Time: 18.04525327682495 Total Reward: -22.53623188405811 Avg_Loss: 0.6390323862903008 Episode: 6: Time: 18.155301094055176 Total Reward: -38.57366771159924 Avg_Loss: 0.7418161563950927 Episode: 7: Time: 18.16043210029602 Total Reward: -30.691318327974717 Avg_Loss: 0.7235426040006285 Episode: 8: Time: 18.129841089248657 Total Reward: 57.77777777778201 Avg_Loss: 0.961006916683762 Episode: 9: Time: 18.364511966705322 Total Reward: 178.35640138408735 Avg_Loss: 1.0793944220332539 Episode: 10: Time: 18.094041109085083 Total Reward: -35.62500000000071 Avg_Loss: 1.2237093282096527 Episode: 11: Time: 17.98344087600708 Total Reward: -32.704918032787575 Avg_Loss: 1.2123449205600916 Episode: 12: Time: 18.02500605583191 Total Reward: -36.37931034482828 Avg_Loss: 1.1815835455385577 Episode: 13: Time: 18.222887992858887 Total Reward: 264.99999999999545 Avg_Loss: 1.4229859458673901 Episode: 14: Time: 18.25511884689331 Total Reward: 221.923076923081 Avg_Loss: 1.7360319133315767 Episode: 15: Time: 18.321141004562378 Total Reward: 315.25641025640334 Avg_Loss: 2.1654331171963395 Episode: 16: Time: 18.624380111694336 Total Reward: 429.6478873239386 Avg_Loss: 2.896785791681594 Episode: 17: Time: 18.144484043121338 Total Reward: 58.33333333333704 Avg_Loss: 2.785179747503345 Episode: 18: Time: 18.347298860549927 Total Reward: 183.93175074184404 Avg_Loss: 2.7864501776815462 Episode: 19: Time: 18.076870918273926 Total Reward: 232.2058823529367 Avg_Loss: 2.806173834730597 Episode: 20: Time: 18.19839906692505 Total Reward: 28.778501628666064 Avg_Loss: 2.7253040706410125 Episode: 21: Time: 18.197052001953125 Total Reward: 301.4912280701668 Avg_Loss: 2.6005463334692624 Episode: 22: Time: 18.17093586921692 Total Reward: 148.5064935064981 Avg_Loss: 2.7995510098813963 Episode: 23: Time: 18.345752000808716 Total Reward: 243.3458646616591 Avg_Loss: 2.7040013320305767 Episode: 24: Time: 18.31086802482605 Total Reward: 306.25391849529353 Avg_Loss: 3.0135432832381306 Episode: 25: Time: 18.190793991088867 Total Reward: 191.8217054263612 Avg_Loss: 2.927862129792446 Episode: 26: Time: 18.086541891098022 Total Reward: 231.00732600732508 Avg_Loss: 3.0213442740320158 Episode: 27: Time: 18.007288217544556 Total Reward: 62.534246575346884 Avg_Loss: 3.2031235614744555 Episode: 28: Time: 18.255620002746582 Total Reward: 365.52631578947006 Avg_Loss: 3.074278678212847 Episode: 29: Time: 18.311011791229248 Total Reward: 351.80851063828504 Avg_Loss: 3.489491711143686 Episode: 30: Time: 18.437655925750732 Total Reward: 408.08641975307705 Avg_Loss: 3.675243974232874 Episode: 31: Time: 18.15780782699585 Total Reward: 243.02816901408954 Avg_Loss: 3.627162521626769 Episode: 32: Time: 18.42415189743042 Total Reward: 237.3262839879113 Avg_Loss: 3.856436508543351 Episode: 33: Time: 18.328477382659912 Total Reward: 462.894736842091 Avg_Loss: 4.038071737569921 Episode: 34: Time: 18.261425971984863 Total Reward: 541.3636363636267 Avg_Loss: 4.097628260610485 Episode: 35: Time: 18.264583110809326 Total Reward: 440.11705685618324 Avg_Loss: 4.326786558668153 Episode: 36: Time: 18.16335105895996 Total Reward: 33.61736334405376 Avg_Loss: 4.386988527634564 Episode: 37: Time: 18.246341228485107 Total Reward: 487.417582417575 Avg_Loss: 4.877088270517958 Episode: 38: Time: 18.2908878326416 Total Reward: 2.402597402596753 Avg_Loss: 5.2357275776502465 Episode: 39: Time: 18.30826497077942 Total Reward: 470.51724137930523 Avg_Loss: 4.848470611231668 Episode: 40: Time: 18.1489999294281 Total Reward: 718.0081300812934 Avg_Loss: 4.797959109314349 Episode: 41: Time: 18.201672077178955 Total Reward: 491.8055555555443 Avg_Loss: 5.223991067469621 Episode: 42: Time: 18.39025115966797 Total Reward: 440.3535353535245 Avg_Loss: 5.30093749431001 Episode: 43: Time: 18.571535110473633 Total Reward: 334.7752808988696 Avg_Loss: 5.413426266998804 Episode: 44: Time: 18.19630193710327 Total Reward: 163.1818181818223 Avg_Loss: 5.199292530532644 Episode: 45: Time: 18.139593839645386 Total Reward: 365.9929078014097 Avg_Loss: 5.687491927327228 Episode: 46: Time: 18.240417003631592 Total Reward: 624.5121951219461 Avg_Loss: 6.073039208139692 Episode: 47: Time: 18.35751986503601 Total Reward: 590.8974358974272 Avg_Loss: 5.887081147241993 Episode: 48: Time: 18.265422105789185 Total Reward: 329.83660130718897 Avg_Loss: 5.79112161107424 Episode: 49: Time: 18.403645992279053 Total Reward: 686.4569536423744 Avg_Loss: 5.773030481418641 Validation Mean Reward: 541.058287179985 Validation Std Reward: 254.27150684373643 Episode: 50: Time: 18.16826605796814 Total Reward: 846.8181818181678 Avg_Loss: 6.759449801525148 Episode: 51: Time: 18.40873098373413 Total Reward: 325.19543973940574 Avg_Loss: 6.149904977373716 Episode: 52: Time: 18.17083501815796 Total Reward: 295.4109589040993 Avg_Loss: 6.317932400382867 Episode: 53: Time: 18.119874954223633 Total Reward: 426.42857142856985 Avg_Loss: 5.801443379967153 Episode: 54: Time: 18.18885898590088 Total Reward: 388.5526315789458 Avg_Loss: 6.268668852433437 Episode: 55: Time: 18.23525881767273 Total Reward: 160.8823529411801 Avg_Loss: 5.931683690608049 Episode: 56: Time: 18.27650213241577 Total Reward: 197.99363057325212 Avg_Loss: 6.604556662695749 Episode: 57: Time: 18.31635308265686 Total Reward: 497.5925925925845 Avg_Loss: 6.751657466928498 Episode: 58: Time: 18.264678716659546 Total Reward: 329.5283018867912 Avg_Loss: 5.8617845002342674 Episode: 59: Time: 18.332592010498047 Total Reward: 436.9865319865245 Avg_Loss: 5.917409225672233 Episode: 60: Time: 18.267860889434814 Total Reward: 216.92660550459158 Avg_Loss: 5.998725859057002 Episode: 61: Time: 18.170814037322998 Total Reward: 366.78343949044023 Avg_Loss: 6.034955018708686 Episode: 62: Time: 18.258047103881836 Total Reward: 297.73927392738796 Avg_Loss: 6.485472918057642 Episode: 63: Time: 18.087783098220825 Total Reward: 473.42105263157066 Avg_Loss: 5.9464718614305765 Episode: 64: Time: 18.179574966430664 Total Reward: 268.9575971731379 Avg_Loss: 5.39176784643606 Episode: 65: Time: 18.251389026641846 Total Reward: 364.016393442617 Avg_Loss: 5.723844048856687 Episode: 66: Time: 18.33535599708557 Total Reward: 299.9044585987231 Avg_Loss: 5.812377886110995 Episode: 67: Time: 18.181712865829468 Total Reward: 413.8339222614812 Avg_Loss: 6.032235279303639 Episode: 68: Time: 18.221739053726196 Total Reward: 299.46366782006817 Avg_Loss: 5.765928756790001 Episode: 69: Time: 18.08717107772827 Total Reward: 324.4756554307012 Avg_Loss: 5.86591976279972 Episode: 70: Time: 18.24286985397339 Total Reward: 605.7874015747974 Avg_Loss: 5.532739667832351 Episode: 71: Time: 17.998521089553833 Total Reward: 40.45816733067724 Avg_Loss: 6.220284319224477 Episode: 72: Time: 18.350736141204834 Total Reward: 200.7317073170758 Avg_Loss: 6.18868879510575 Episode: 73: Time: 18.324075937271118 Total Reward: 162.66871165644568 Avg_Loss: 5.796672376264043 Episode: 74: Time: 18.330111980438232 Total Reward: 157.77777777778073 Avg_Loss: 5.6299952478969795 Episode: 75: Time: 18.291900157928467 Total Reward: 401.5517241379263 Avg_Loss: 5.911153344046168 Episode: 76: Time: 18.32991123199463 Total Reward: 261.6666666666628 Avg_Loss: 5.724857014768264 Episode: 77: Time: 18.459070920944214 Total Reward: 299.02985074626343 Avg_Loss: 5.26887596404853 Episode: 78: Time: 18.304823875427246 Total Reward: 491.44067796609636 Avg_Loss: 5.449790152681976 Episode: 79: Time: 18.072685956954956 Total Reward: 220.57377049180158 Avg_Loss: 5.732555697445108 Episode: 80: Time: 18.39275312423706 Total Reward: 228.5294117647092 Avg_Loss: 6.182037453691499 Episode: 81: Time: 18.282240867614746 Total Reward: 296.0256410256374 Avg_Loss: 5.646013037497256 Episode: 82: Time: 18.211884021759033 Total Reward: 201.6666666666681 Avg_Loss: 5.39023963793987 Episode: 83: Time: 18.212878942489624 Total Reward: 488.0388692579453 Avg_Loss: 5.397463329198982
Episode: 84: Time: 18.142138242721558 Total Reward: 416.9999999999958 Avg_Loss: 5.612673391814993 Episode: 85: Time: 18.4061918258667 Total Reward: 319.492753623184 Avg_Loss: 5.195713648275167 Episode: 86: Time: 18.25415301322937 Total Reward: 528.2394366197111 Avg_Loss: 6.193075724509584 Episode: 87: Time: 18.327216863632202 Total Reward: 343.4858044164013 Avg_Loss: 5.504364591185786 Episode: 88: Time: 18.179681062698364 Total Reward: 359.5454545454494 Avg_Loss: 5.4503538037548545 Episode: 89: Time: 18.26184606552124 Total Reward: 5.977198697067816 Avg_Loss: 5.654433987721675 Episode: 90: Time: 18.217226028442383 Total Reward: 180.98566308244153 Avg_Loss: 5.519748319096926 Episode: 91: Time: 18.269148111343384 Total Reward: 444.2857142857073 Avg_Loss: 5.6215470127698755 Episode: 92: Time: 18.24690008163452 Total Reward: 270.5172413793123 Avg_Loss: 5.845609871780171 Episode: 93: Time: 18.235082864761353 Total Reward: 356.72413793103215 Avg_Loss: 5.225014079017799 Episode: 94: Time: 18.03082275390625 Total Reward: 210.55555555556037 Avg_Loss: 5.141586678869584 Episode: 95: Time: 18.115129232406616 Total Reward: 136.66023166023254 Avg_Loss: 5.797506736106231 Episode: 96: Time: 18.107945919036865 Total Reward: 209.02930402930565 Avg_Loss: 5.710746499670654 Episode: 97: Time: 18.33087706565857 Total Reward: 200.68106312292838 Avg_Loss: 5.667083384610024 Episode: 98: Time: 18.24197030067444 Total Reward: 198.10344827586502 Avg_Loss: 5.555940789334914 Episode: 99: Time: 18.076009035110474 Total Reward: 310.40540540540155 Avg_Loss: 5.683594487795309 Validation Mean Reward: -90.78204298374992 Validation Std Reward: 3.1669149300825508 Episode: 100: Time: 20.567848205566406 Total Reward: 269.4859813084112 Avg_Loss: 5.8256769290491315 Episode: 101: Time: 19.089650869369507 Total Reward: 212.16723549488518 Avg_Loss: 5.47114548162252 Episode: 102: Time: 20.38584589958191 Total Reward: 252.82608695652635 Avg_Loss: 6.139193460720928 Episode: 103: Time: 18.93939995765686 Total Reward: 41.98630136986425 Avg_Loss: 6.22513021441067 Episode: 104: Time: 18.40470600128174 Total Reward: 445.6504065040593 Avg_Loss: 5.58875932412989 Episode: 105: Time: 18.426018953323364 Total Reward: 264.0504451038606 Avg_Loss: 6.149070255896625 Episode: 106: Time: 18.112996101379395 Total Reward: 337.1428571428493 Avg_Loss: 5.686606180768053 Episode: 107: Time: 18.199383974075317 Total Reward: 388.3948339483369 Avg_Loss: 6.391099219562626 Episode: 108: Time: 17.843034982681274 Total Reward: 377.2222222222163 Avg_Loss: 6.109251760635056 Episode: 109: Time: 18.00417399406433 Total Reward: 440.5805243445633 Avg_Loss: 5.950793890392079 Episode: 110: Time: 18.057311058044434 Total Reward: 485.07117437721786 Avg_Loss: 6.511567727858279 Episode: 111: Time: 18.04679298400879 Total Reward: 469.5756457564539 Avg_Loss: 5.92134409792283 Episode: 112: Time: 18.02759885787964 Total Reward: 393.2943143812678 Avg_Loss: 6.240048531724625 Episode: 113: Time: 18.004342079162598 Total Reward: 446.25412541253513 Avg_Loss: 6.0937060338108475 Episode: 114: Time: 18.21419405937195 Total Reward: 375.2194357366743 Avg_Loss: 5.7862304919908025 Episode: 115: Time: 17.951839923858643 Total Reward: 439.24657534246063 Avg_Loss: 6.157321171600278 Episode: 116: Time: 17.89620804786682 Total Reward: 313.93470790377916 Avg_Loss: 6.307279964455035 Episode: 117: Time: 18.172892093658447 Total Reward: 272.81609195402143 Avg_Loss: 5.930768018510161 Episode: 118: Time: 17.898707151412964 Total Reward: 493.4773662551387 Avg_Loss: 5.908713221549988 Episode: 119: Time: 17.871718883514404 Total Reward: 355.18450184501705 Avg_Loss: 6.0961543692260225 Episode: 120: Time: 18.097790241241455 Total Reward: 306.27388535031656 Avg_Loss: 6.20575611982025 Episode: 121: Time: 18.145498037338257 Total Reward: 341.36363636362694 Avg_Loss: 5.72462134701865 Episode: 122: Time: 17.9094021320343 Total Reward: 509.16666666665947 Avg_Loss: 6.319949044900782 Episode: 123: Time: 17.93363094329834 Total Reward: 383.87323943661767 Avg_Loss: 5.835614162332871 Episode: 124: Time: 17.83166193962097 Total Reward: 370.70397111913195 Avg_Loss: 6.403026789176364 Episode: 125: Time: 17.924437999725342 Total Reward: 223.49315068493377 Avg_Loss: 6.320720907018966 Episode: 126: Time: 17.971867322921753 Total Reward: 191.16352201258195 Avg_Loss: 5.845471500348644 Episode: 127: Time: 17.846501111984253 Total Reward: 66.65413533835014 Avg_Loss: 5.683196052783678 Episode: 128: Time: 17.99811887741089 Total Reward: 199.11764705882703 Avg_Loss: 6.386481841071313 Episode: 129: Time: 17.91674017906189 Total Reward: 232.58620689655567 Avg_Loss: 6.3843616978461 Episode: 130: Time: 17.965538024902344 Total Reward: 364.8540145985311 Avg_Loss: 6.010565348032142 Episode: 131: Time: 18.111664056777954 Total Reward: 218.39031339031595 Avg_Loss: 6.365950674569907 Episode: 132: Time: 17.960154056549072 Total Reward: 440.58052434456425 Avg_Loss: 6.202221132126175 Episode: 133: Time: 17.919108152389526 Total Reward: 137.48407643312447 Avg_Loss: 6.807804863993861 Episode: 134: Time: 17.89201807975769 Total Reward: 395.1315789473664 Avg_Loss: 6.66185387242742 Episode: 135: Time: 17.84021306037903 Total Reward: 361.4459930313571 Avg_Loss: 6.575222785733327 Episode: 136: Time: 17.791788816452026 Total Reward: -17.22222222222297 Avg_Loss: 6.157482479800697 Episode: 137: Time: 18.013685703277588 Total Reward: 470.21739130433855 Avg_Loss: 6.8249893148406215 Episode: 138: Time: 17.840290069580078 Total Reward: 352.76119402984665 Avg_Loss: 6.550146355348475 Episode: 139: Time: 17.970367908477783 Total Reward: 187.39202657807766 Avg_Loss: 6.320237037514438 Episode: 140: Time: 18.109021186828613 Total Reward: 177.41379310345073 Avg_Loss: 5.98322209089744 Episode: 141: Time: 18.245090007781982 Total Reward: 207.63157894737208 Avg_Loss: 6.549756331604068 Episode: 142: Time: 18.08811616897583 Total Reward: 246.91176470588357 Avg_Loss: 6.531976585628605 Episode: 143: Time: 17.91150712966919 Total Reward: 507.9962546816406 Avg_Loss: 6.728533272983647 Episode: 144: Time: 18.057901859283447 Total Reward: 344.9999999999927 Avg_Loss: 6.4382487715793255 Episode: 145: Time: 17.917963981628418 Total Reward: 342.26235741444924 Avg_Loss: 6.20372994006181 Episode: 146: Time: 17.91820502281189 Total Reward: 113.60927152318268 Avg_Loss: 7.031043098754242 Episode: 147: Time: 17.990538120269775 Total Reward: 377.72727272727036 Avg_Loss: 6.620708621349655 Episode: 148: Time: 18.1126549243927 Total Reward: 297.8571428571406 Avg_Loss: 6.736689405281003 Episode: 149: Time: 18.099767923355103 Total Reward: 178.9726027397297 Avg_Loss: 6.545551816956336 Validation Mean Reward: 514.9972693107807 Validation Std Reward: 140.78194782275878 Episode: 150: Time: 18.39292597770691 Total Reward: 382.5086505190276 Avg_Loss: 6.451722028876553 Episode: 151: Time: 18.271579027175903 Total Reward: 312.53424657534293 Avg_Loss: 6.6661014121119715 Episode: 152: Time: 18.126032829284668 Total Reward: 432.8810408921903 Avg_Loss: 6.985967873525219 Episode: 153: Time: 18.076515674591064 Total Reward: 294.6551724137931 Avg_Loss: 6.7571075964374705 Episode: 154: Time: 17.98317813873291 Total Reward: 233.85906040268807 Avg_Loss: 6.3841343396851995 Episode: 155: Time: 17.953184127807617 Total Reward: 507.3622047243998 Avg_Loss: 6.166478180083908 Episode: 156: Time: 17.870534896850586 Total Reward: 492.0445344129458 Avg_Loss: 7.171737905309982 Episode: 157: Time: 18.007779121398926 Total Reward: 446.51624548735464 Avg_Loss: 6.804711107446366 Episode: 158: Time: 18.00493621826172 Total Reward: 326.05263157894467 Avg_Loss: 6.330253730301096 Episode: 159: Time: 17.878777742385864 Total Reward: 10.084745762712718 Avg_Loss: 7.001849071318362 Episode: 160: Time: 17.79886293411255 Total Reward: 428.07692307691826 Avg_Loss: 7.16655613093817 Episode: 161: Time: 17.983182668685913 Total Reward: 368.022508038581 Avg_Loss: 6.455763737694556 Episode: 162: Time: 18.258513689041138 Total Reward: 109.4025157232748 Avg_Loss: 7.17879484781698 Episode: 163: Time: 18.255913972854614 Total Reward: 217.7035830618938 Avg_Loss: 6.343413259802746 Episode: 164: Time: 18.426661729812622 Total Reward: 178.61563517915408 Avg_Loss: 6.740041036565764 Episode: 165: Time: 18.183462142944336 Total Reward: 413.9605734767003 Avg_Loss: 7.161009117334831
Episode: 166: Time: 17.96447205543518 Total Reward: 132.94117647059218 Avg_Loss: 6.775438674357759 Episode: 167: Time: 18.11350417137146 Total Reward: 348.66197183098706 Avg_Loss: 7.16400932464279 Episode: 168: Time: 18.020145893096924 Total Reward: 304.3506493506491 Avg_Loss: 6.943542195969269 Episode: 169: Time: 18.52626395225525 Total Reward: 406.5479876160962 Avg_Loss: 6.4173817043544865 Episode: 170: Time: 18.629361867904663 Total Reward: 329.1379310344759 Avg_Loss: 6.437998766658687 Episode: 171: Time: 18.5725998878479 Total Reward: 455.4885993485277 Avg_Loss: 7.113273433276585 Episode: 172: Time: 18.536601066589355 Total Reward: 365.8150470219387 Avg_Loss: 7.059104294336143 Episode: 173: Time: 18.406563997268677 Total Reward: 423.9003436426087 Avg_Loss: 6.942892452248004 Episode: 174: Time: 18.16997194290161 Total Reward: 325.13888888889 Avg_Loss: 6.5807209024910165 Episode: 175: Time: 18.126417875289917 Total Reward: 292.4538745387388 Avg_Loss: 7.062607043430585 Episode: 176: Time: 18.11974811553955 Total Reward: 369.88294314380823 Avg_Loss: 6.7255358185086935 Episode: 177: Time: 18.025427103042603 Total Reward: 421.0142348754392 Avg_Loss: 6.295909025088078 Episode: 178: Time: 18.085229873657227 Total Reward: 380.40983606557006 Avg_Loss: 6.790554531482088 Episode: 179: Time: 18.05420422554016 Total Reward: 514.665427509284 Avg_Loss: 6.217391631182502 Episode: 180: Time: 18.01965618133545 Total Reward: 529.5353159851239 Avg_Loss: 6.854352847868655 Episode: 181: Time: 17.92475199699402 Total Reward: 425.29520295202724 Avg_Loss: 6.17097666684319 Episode: 182: Time: 17.879291772842407 Total Reward: 498.0232558139478 Avg_Loss: 6.42788399568125 Episode: 183: Time: 18.04137897491455 Total Reward: 397.857142857137 Avg_Loss: 7.316728614959397 Episode: 184: Time: 17.808637857437134 Total Reward: 299.53124999999227 Avg_Loss: 7.6655492411942046 Episode: 185: Time: 18.191431045532227 Total Reward: 424.4346289752615 Avg_Loss: 6.930419908852136 Episode: 186: Time: 18.291975021362305 Total Reward: 330.9927797833896 Avg_Loss: 6.830522296809349 Episode: 187: Time: 18.278334140777588 Total Reward: 290.3820598006644 Avg_Loss: 7.030155645198181 Episode: 188: Time: 18.196964979171753 Total Reward: 245.98360655737602 Avg_Loss: 6.864343808478668 Episode: 189: Time: 18.29747200012207 Total Reward: 144.87538940810404 Avg_Loss: 7.333124545441956 Episode: 190: Time: 18.289777994155884 Total Reward: 594.7810218977996 Avg_Loss: 6.423504394643447 Episode: 191: Time: 18.27003002166748 Total Reward: 299.3217665615132 Avg_Loss: 7.0687377292568945 Episode: 192: Time: 18.12866711616516 Total Reward: 136.57894736842343 Avg_Loss: 7.636868269503617 Episode: 193: Time: 18.033663988113403 Total Reward: 266.0223642172445 Avg_Loss: 6.534649846934471 Episode: 194: Time: 18.055454969406128 Total Reward: 384.3103448275761 Avg_Loss: 6.727145845148744 Episode: 195: Time: 18.30987310409546 Total Reward: 409.58715596329967 Avg_Loss: 6.949862620409797 Episode: 196: Time: 18.1718647480011 Total Reward: 272.7419354838722 Avg_Loss: 6.849894391388452 Episode: 197: Time: 17.99349093437195 Total Reward: 417.2807017543772 Avg_Loss: 6.730232763691109 Episode: 198: Time: 18.19780707359314 Total Reward: 255.45317220544007 Avg_Loss: 6.914558271400067 Episode: 199: Time: 17.918556928634644 Total Reward: 510.7347670250816 Avg_Loss: 7.603130750295494 Validation Mean Reward: 540.2062562942355 Validation Std Reward: 219.69247805641848 Test Mean Reward: 496.58674988644236 Test Std Reward: 262.158061334461
xxxxxxxxxxtrainerDQN2 = DQN.DQN(EnvWrapper(env), model.Nature_Paper_Conv, lr = 0.00025, gamma = 0.95, buffer_size=100000, batch_size=32, loss_fn = "smooth_l1_loss", use_wandb = False, device = 'cpu', seed = 42, epsilon_scheduler = utils.exponential_decay(1, 700,0.1), save_path = utils.get_save_path("DQN","./runs/"))out_l1 = trainerDQN2.train(200,50,30,50,50) saving to ./runs/DQN/run1 Episode: 1: Time: 13.697196960449219 Total Reward: -40.578231292517756 Avg_Loss: 0.13669895916690777 Episode: 2: Time: 18.738681077957153 Total Reward: -66.53024911032085 Avg_Loss: 0.14641240352428198 Episode: 3: Time: 18.74876308441162 Total Reward: -71.02739726027448 Avg_Loss: 0.11805925721365239 Episode: 4: Time: 18.71462082862854 Total Reward: -82.26114649681492 Avg_Loss: 0.10940684976463183 Episode: 5: Time: 18.661263942718506 Total Reward: -59.705882352941636 Avg_Loss: 0.10656297952621518 Episode: 6: Time: 18.547620058059692 Total Reward: -56.26760563380327 Avg_Loss: 0.11676109245070453 Episode: 7: Time: 18.43413209915161 Total Reward: -84.09090909090872 Avg_Loss: 0.11963350077241879 Episode: 8: Time: 18.308965921401978 Total Reward: -29.853420195440265 Avg_Loss: 0.11629590964602197 Episode: 9: Time: 17.528201818466187 Total Reward: -103.38791208791196 Avg_Loss: 0.12750743260082492 Episode: 10: Time: 18.157769203186035 Total Reward: -26.407942238267864 Avg_Loss: 0.16890396736953564 Episode: 11: Time: 18.20663022994995 Total Reward: -72.41935483870989 Avg_Loss: 0.20273878035221657 Episode: 12: Time: 18.08461308479309 Total Reward: -28.33333333333404 Avg_Loss: 0.2000100773234828 Episode: 13: Time: 18.357059001922607 Total Reward: -31.026936026936614 Avg_Loss: 0.18970570681291876 Episode: 14: Time: 4.353758811950684 Total Reward: -61.16655290102411 Avg_Loss: 0.20905740671023204 Episode: 15: Time: 18.375272750854492 Total Reward: -16.768707482993893 Avg_Loss: 0.26216258674453036 Episode: 16: Time: 18.16376495361328 Total Reward: -25.693069306931438 Avg_Loss: 0.21202245092082048 Episode: 17: Time: 18.17576766014099 Total Reward: -23.32764505119524 Avg_Loss: 0.24185364960278033 Episode: 18: Time: 15.007147073745728 Total Reward: -14.464835164831925 Avg_Loss: 0.24080933088988965 Episode: 19: Time: 17.72961115837097 Total Reward: -32.71626297577926 Avg_Loss: 0.3190385792013101 Episode: 20: Time: 17.826109886169434 Total Reward: -23.082191780822587 Avg_Loss: 0.3482424097073053 Episode: 21: Time: 17.82283306121826 Total Reward: -26.97278911564694 Avg_Loss: 0.2513860042300858 Episode: 22: Time: 18.017571210861206 Total Reward: 48.302180685361456 Avg_Loss: 0.3428701682157126 Episode: 23: Time: 17.705981969833374 Total Reward: -22.631578947369096 Avg_Loss: 0.3281545343362734 Episode: 24: Time: 17.905699014663696 Total Reward: -19.812030075188556 Avg_Loss: 0.30799881562183634 Episode: 25: Time: 15.840013027191162 Total Reward: -115.32456140350942 Avg_Loss: 0.42202760382755616 Episode: 26: Time: 17.836683988571167 Total Reward: -33.27160493827226 Avg_Loss: 0.34761271073224664 Episode: 27: Time: 17.935446977615356 Total Reward: 127.22222222222584 Avg_Loss: 0.3506204451457793 Episode: 28: Time: 17.871068954467773 Total Reward: -23.33876221498438 Avg_Loss: 0.3977863444093646 Episode: 29: Time: 10.750377893447876 Total Reward: -48.089108910889415 Avg_Loss: 0.31595773710966946 Episode: 30: Time: 17.74188995361328 Total Reward: -32.304075235110446 Avg_Loss: 0.3654998705542388 Episode: 31: Time: 17.761902809143066 Total Reward: -33.84892086331007 Avg_Loss: 0.34844386141684625 Episode: 32: Time: 17.983778715133667 Total Reward: -20.373134328358848 Avg_Loss: 0.3834315945057809 Episode: 33: Time: 18.158904790878296 Total Reward: -26.596091205212385 Avg_Loss: 0.35838024599962875 Episode: 34: Time: 17.58035898208618 Total Reward: 41.17021276596073 Avg_Loss: 0.36707023601634664 Episode: 35: Time: 17.804464101791382 Total Reward: -32.71626297577926 Avg_Loss: 0.31764011205557513 Episode: 36: Time: 17.88987112045288 Total Reward: -39.11764705882423 Avg_Loss: 0.35640697035656754 Episode: 37: Time: 17.86575698852539 Total Reward: -31.50793650793701 Avg_Loss: 0.35509143948570643 Episode: 38: Time: 17.617708921432495 Total Reward: -37.85714285714353 Avg_Loss: 0.36808083563291727 Episode: 39: Time: 17.939188957214355 Total Reward: -24.712460063898483 Avg_Loss: 0.3263932291199179 Episode: 40: Time: 17.83411693572998 Total Reward: 22.02127659574615 Avg_Loss: 0.3547352159849736 Episode: 41: Time: 17.738868951797485 Total Reward: -28.554817275748256 Avg_Loss: 0.32368412572389643 Episode: 42: Time: 17.074321031570435 Total Reward: -86.32977099236527 Avg_Loss: 0.3945632729456465 Episode: 43: Time: 18.07896399497986 Total Reward: 163.46153846154235 Avg_Loss: 0.4009094623019214 Episode: 44: Time: 17.754206895828247 Total Reward: 131.0536398467443 Avg_Loss: 0.3691764635448696 Episode: 45: Time: 17.84895396232605 Total Reward: 26.107266435986276 Avg_Loss: 0.3924787211537111 Episode: 46: Time: 17.867650032043457 Total Reward: 95.11406844106892 Avg_Loss: 0.4217574196640684 Episode: 47: Time: 17.95916986465454 Total Reward: 150.67474048443344 Avg_Loss: 0.45597091968319997 Episode: 48: Time: 17.989299297332764 Total Reward: 376.3804713804715 Avg_Loss: 0.41088835639926063 Episode: 49: Time: 18.1018967628479 Total Reward: 193.52459016393894 Avg_Loss: 0.5111667192658457 Validation Mean Reward: 128.91742714437638 Validation Std Reward: 81.58430482956773 Episode: 50: Time: 17.84545087814331 Total Reward: 128.04832713754706 Avg_Loss: 0.5572973694965369 Episode: 51: Time: 17.98436689376831 Total Reward: 43.047138047141054 Avg_Loss: 0.6398262293430186 Episode: 52: Time: 18.03160071372986 Total Reward: 197.41877256318105 Avg_Loss: 0.6309304560745964 Episode: 53: Time: 17.833609104156494 Total Reward: 28.07692307692502 Avg_Loss: 0.6414437536932841 Episode: 54: Time: 17.822794914245605 Total Reward: 165.2739726027441 Avg_Loss: 0.6526275279016054 Episode: 55: Time: 18.03430414199829 Total Reward: 148.58974358974788 Avg_Loss: 0.6814537403338096 Episode: 56: Time: 18.095867156982422 Total Reward: 133.75816993464244 Avg_Loss: 0.7007359606879098 Episode: 57: Time: 18.131829977035522 Total Reward: 320.5844155844146 Avg_Loss: 0.678916985622975 Episode: 58: Time: 18.026703119277954 Total Reward: 315.52631578947114 Avg_Loss: 0.7358217913688732 Episode: 59: Time: 17.838567972183228 Total Reward: 100.12195121951669 Avg_Loss: 0.7652317851844455 Episode: 60: Time: 17.88171911239624 Total Reward: 34.23076923077148 Avg_Loss: 0.7543396876514459 Episode: 61: Time: 17.833049058914185 Total Reward: 64.3220338983096 Avg_Loss: 0.7552861023475143 Episode: 62: Time: 17.765434980392456 Total Reward: 195.5660377358539 Avg_Loss: 0.7955300342761168 Episode: 63: Time: 17.74130606651306 Total Reward: 206.00334448161007 Avg_Loss: 0.7428282997688326 Episode: 64: Time: 18.02334690093994 Total Reward: 178.9273927392773 Avg_Loss: 0.8380446814063216 Episode: 65: Time: 17.892439126968384 Total Reward: 23.589743589744188 Avg_Loss: 0.7825195307866866 Episode: 66: Time: 17.810699224472046 Total Reward: 251.1538461538366 Avg_Loss: 0.8052860626152584 Episode: 67: Time: 17.87334394454956 Total Reward: 401.29629629628937 Avg_Loss: 0.8467484224243325 Episode: 68: Time: 18.117341995239258 Total Reward: 37.25806451613128 Avg_Loss: 0.9124913082904175 Episode: 69: Time: 17.849833011627197 Total Reward: 428.99999999999307 Avg_Loss: 0.908911964723042 Episode: 70: Time: 17.92949676513672 Total Reward: 283.2894736842104 Avg_Loss: 0.8834309448965457 Episode: 71: Time: 19.12355399131775 Total Reward: 434.20962199311975 Avg_Loss: 0.9208756507444782 Episode: 72: Time: 17.9132559299469 Total Reward: 40.22012578616579 Avg_Loss: 0.9097590288695168 Episode: 73: Time: 18.01451086997986 Total Reward: 242.31343283582424 Avg_Loss: 0.9548225993243586 Episode: 74: Time: 17.896542072296143 Total Reward: 447.0875420875348 Avg_Loss: 0.9789732347766892 Episode: 75: Time: 17.935574054718018 Total Reward: 268.63636363635896 Avg_Loss: 0.9982016114126734 Episode: 76: Time: 17.64147400856018 Total Reward: 144.04382470119964 Avg_Loss: 1.0018143894291724 Episode: 77: Time: 17.897335052490234 Total Reward: 335.5084745762651 Avg_Loss: 1.0095091741375561 Episode: 78: Time: 17.76966094970703 Total Reward: 350.2554744525433 Avg_Loss: 0.9987056603451737 Episode: 79: Time: 17.92498517036438 Total Reward: 235.15873015873387 Avg_Loss: 0.9998028482459173 Episode: 80: Time: 17.72872304916382 Total Reward: 264.32203389830846 Avg_Loss: 1.0281140214505315 Episode: 81: Time: 17.762802839279175 Total Reward: 307.47678018575846 Avg_Loss: 1.0456956447172565 Episode: 82: Time: 17.76514983177185 Total Reward: 297.36111111110154 Avg_Loss: 1.0458453515497577
Episode: 83: Time: 17.68531608581543 Total Reward: 502.2222222222158 Avg_Loss: 1.0311509353273056 Episode: 84: Time: 17.619313955307007 Total Reward: 185.44280442804873 Avg_Loss: 1.027345991560391 Episode: 85: Time: 17.760406017303467 Total Reward: 496.69550173009725 Avg_Loss: 1.08883251387532 Episode: 86: Time: 17.98920512199402 Total Reward: 289.32835820895696 Avg_Loss: 1.0815375089144506 Episode: 87: Time: 17.930089950561523 Total Reward: 356.7374517374443 Avg_Loss: 1.0414030373472125 Episode: 88: Time: 18.195363998413086 Total Reward: 380.0830564783992 Avg_Loss: 1.1424517409891641 Episode: 89: Time: 18.121834993362427 Total Reward: 412.5187969924686 Avg_Loss: 1.118601833446687 Episode: 90: Time: 18.227668046951294 Total Reward: 382.8481012658179 Avg_Loss: 1.1771620900190176 Episode: 91: Time: 18.027714014053345 Total Reward: 269.9635036496382 Avg_Loss: 1.142725749676969 Episode: 92: Time: 18.15070104598999 Total Reward: 556.7241379310241 Avg_Loss: 1.2236027463394052 Episode: 93: Time: 17.986899852752686 Total Reward: 403.33887043188986 Avg_Loss: 1.2073113394885504 Episode: 94: Time: 18.069560050964355 Total Reward: 232.58620689655555 Avg_Loss: 1.2216551937475926 Episode: 95: Time: 18.110829830169678 Total Reward: 364.7315436241608 Avg_Loss: 1.117572479513513 Episode: 96: Time: 18.05068325996399 Total Reward: 172.7966101694929 Avg_Loss: 1.239300388122807 Episode: 97: Time: 20.551268815994263 Total Reward: 361.52173913042463 Avg_Loss: 1.172255143648436 Episode: 98: Time: 23.73601984977722 Total Reward: 157.5951557093466 Avg_Loss: 1.1982765426906217 Episode: 99: Time: 23.678724765777588 Total Reward: 317.21374045800826 Avg_Loss: 1.3078460796039646 Validation Mean Reward: 451.77004493431707 Validation Std Reward: 145.94292225376492 Episode: 100: Time: 3961.143696308136 Total Reward: 228.89937106918666 Avg_Loss: 1.2338739425695242 Episode: 101: Time: 3710.984722852707 Total Reward: 361.1403508771854 Avg_Loss: 1.165178513952664 Episode: 102: Time: 1902.8415801525116 Total Reward: 190.71428571429027 Avg_Loss: 1.2823090072439498 Episode: 103: Time: 721.1812899112701 Total Reward: 440.3159851301001 Avg_Loss: 1.3539434779091042 Episode: 104: Time: 23.95649290084839 Total Reward: 480.86206896550937 Avg_Loss: 1.2415425568318166 Episode: 105: Time: 23.91793990135193 Total Reward: 415.34482758620214 Avg_Loss: 1.2725451667519176 Episode: 106: Time: 23.722827911376953 Total Reward: 474.8924731182731 Avg_Loss: 1.2797850347867532 Episode: 107: Time: 23.841699838638306 Total Reward: 195.54054054054103 Avg_Loss: 1.2122203121666146 Episode: 108: Time: 23.698938131332397 Total Reward: 290.1132686084042 Avg_Loss: 1.1960050032920195 Episode: 109: Time: 23.703282117843628 Total Reward: 172.7419354838743 Avg_Loss: 1.2294208547648262 Episode: 110: Time: 23.487534046173096 Total Reward: 341.61971830985857 Avg_Loss: 1.2370800292041122 Episode: 111: Time: 23.63881492614746 Total Reward: 183.28746177370482 Avg_Loss: 1.3482507945609694 Episode: 112: Time: 23.466731071472168 Total Reward: 471.78700361010175 Avg_Loss: 1.2846375666245693 Episode: 113: Time: 23.482064247131348 Total Reward: 503.6622073578524 Avg_Loss: 1.2918039804496686 Episode: 114: Time: 23.428942918777466 Total Reward: 339.4569288389488 Avg_Loss: 1.2927496813926376 Episode: 115: Time: 23.5871901512146 Total Reward: 459.7945205479373 Avg_Loss: 1.2964004501074302 Episode: 116: Time: 23.710290908813477 Total Reward: 467.3003194888115 Avg_Loss: 1.2986431988347478 Episode: 117: Time: 23.50993514060974 Total Reward: 357.961672473864 Avg_Loss: 1.3174566801856546 Episode: 118: Time: 23.44378924369812 Total Reward: 388.33333333332996 Avg_Loss: 1.3120598752959436 Episode: 119: Time: 23.617539882659912 Total Reward: 441.6666666666587 Avg_Loss: 1.2437233767088722 Episode: 120: Time: 23.56581211090088 Total Reward: 508.508771929814 Avg_Loss: 1.3612042800718998 Episode: 121: Time: 23.510443925857544 Total Reward: 567.9629629629535 Avg_Loss: 1.3122411755954517 Episode: 122: Time: 23.5451397895813 Total Reward: 401.6666666666633 Avg_Loss: 1.3427824183922856 Episode: 123: Time: 23.401777029037476 Total Reward: 540.2941176470495 Avg_Loss: 1.3492610537204421 Episode: 124: Time: 23.61142587661743 Total Reward: 356.82724252491033 Avg_Loss: 1.3761317704905982 Episode: 125: Time: 23.667736768722534 Total Reward: 106.38888888889372 Avg_Loss: 1.3386931274117542 Episode: 126: Time: 23.530205011367798 Total Reward: 513.9965397923781 Avg_Loss: 1.3757133138279956 Episode: 127: Time: 23.61895179748535 Total Reward: 334.8780487804814 Avg_Loss: 1.3439647507767718 Episode: 128: Time: 23.50975775718689 Total Reward: 329.52830188678166 Avg_Loss: 1.3099992530686515 Episode: 129: Time: 23.759350061416626 Total Reward: 334.4871794871685 Avg_Loss: 1.3408742922694743 Episode: 130: Time: 23.58913803100586 Total Reward: 310.7507987220333 Avg_Loss: 1.3491808656884843 Episode: 131: Time: 23.485838890075684 Total Reward: 369.6840148698796 Avg_Loss: 1.3834956878123164 Episode: 132: Time: 23.66059398651123 Total Reward: 234.1139240506381 Avg_Loss: 1.2911015990651955 Episode: 133: Time: 23.620940685272217 Total Reward: 337.98969072165005 Avg_Loss: 1.3085551144195204 Episode: 134: Time: 23.49816608428955 Total Reward: 453.8215488215385 Avg_Loss: 1.3701591904924697 Episode: 135: Time: 23.741374015808105 Total Reward: 485.44164037853983 Avg_Loss: 1.4522322771929894 Episode: 136: Time: 23.71074414253235 Total Reward: 497.35668789808125 Avg_Loss: 1.3636324603517516 Episode: 137: Time: 23.487149953842163 Total Reward: 373.43853820597747 Avg_Loss: 1.3123780526533848 Episode: 138: Time: 23.438434839248657 Total Reward: 509.89510489509865 Avg_Loss: 1.4391781462340796 Episode: 139: Time: 23.698197841644287 Total Reward: 489.90566037734607 Avg_Loss: 1.453561236878403 Episode: 140: Time: 23.501288890838623 Total Reward: 579.7404844290577 Avg_Loss: 1.4683193704661202 Episode: 141: Time: 23.551772356033325 Total Reward: 380.5244755244691 Avg_Loss: 1.4239216033651048 Episode: 142: Time: 23.467719078063965 Total Reward: 565.7142857142718 Avg_Loss: 1.4595711934967202 Episode: 143: Time: 23.539000749588013 Total Reward: 410.01672240802316 Avg_Loss: 1.4684817748410361 Episode: 144: Time: 23.55061388015747 Total Reward: 242.0786516853971 Avg_Loss: 1.4922592810222082 Episode: 145: Time: 23.64220905303955 Total Reward: 317.69841269840214 Avg_Loss: 1.4593577445054255 Episode: 146: Time: 23.41742515563965 Total Reward: 582.9026217228378 Avg_Loss: 1.3778035806006743 Episode: 147: Time: 23.472739219665527 Total Reward: 517.4567474048354 Avg_Loss: 1.444518752959596 Episode: 148: Time: 23.60138988494873 Total Reward: 596.1764705882267 Avg_Loss: 1.412740967353853 Episode: 149: Time: 23.561918020248413 Total Reward: 141.30136986301815 Avg_Loss: 1.3774528831493955 Validation Mean Reward: 480.4727367149716 Validation Std Reward: 152.49769788279212 Episode: 150: Time: 23.470022201538086 Total Reward: 481.7790262172223 Avg_Loss: 1.488356172036724 Episode: 151: Time: 23.482831239700317 Total Reward: 479.9999999999924 Avg_Loss: 1.4946973454050656 Episode: 152: Time: 23.533387899398804 Total Reward: 681.1732851985479 Avg_Loss: 1.5307431624216192 Episode: 153: Time: 23.518216133117676 Total Reward: 435.20134228187385 Avg_Loss: 1.510216445231638 Episode: 154: Time: 23.40952205657959 Total Reward: 401.62162162161627 Avg_Loss: 1.5083578947712393 Episode: 155: Time: 23.509818077087402 Total Reward: 411.80272108842826 Avg_Loss: 1.4743906156856472 Episode: 156: Time: 23.425094842910767 Total Reward: 481.64233576641766 Avg_Loss: 1.5082673172489935 Episode: 157: Time: 23.425482034683228 Total Reward: 594.2857142857054 Avg_Loss: 1.5008161734132206 Episode: 158: Time: 23.436623096466064 Total Reward: 303.7138263665484 Avg_Loss: 1.5338238695589435 Episode: 159: Time: 23.4198899269104 Total Reward: 814.803921568612 Avg_Loss: 1.4681759727101367 Episode: 160: Time: 23.395944118499756 Total Reward: 625.5387205387095 Avg_Loss: 1.67580811892237 Episode: 161: Time: 23.44347620010376 Total Reward: 478.37883959043074 Avg_Loss: 1.5535604267561136 Episode: 162: Time: 23.496777772903442 Total Reward: 374.59459459458895 Avg_Loss: 1.5930095114627807 Episode: 163: Time: 23.571472883224487 Total Reward: 225.12195121951345 Avg_Loss: 1.5926946213766306
Episode: 164: Time: 23.42771005630493 Total Reward: 567.9213483145977 Avg_Loss: 1.6404050002578927 Episode: 165: Time: 23.712730169296265 Total Reward: 452.5409836065482 Avg_Loss: 1.6160642619894332 Episode: 166: Time: 23.41264581680298 Total Reward: 682.385159010588 Avg_Loss: 1.4927846526398378 Episode: 167: Time: 23.340200185775757 Total Reward: 638.8129496402737 Avg_Loss: 1.5835837125778198 Episode: 168: Time: 23.429304122924805 Total Reward: 556.5679442508614 Avg_Loss: 1.4214394851392056 Episode: 169: Time: 23.538545846939087 Total Reward: 615.8843537414891 Avg_Loss: 1.6086778660782246 Episode: 170: Time: 23.766814947128296 Total Reward: 427.59887005648966 Avg_Loss: 1.6265828679589664 Episode: 171: Time: 23.571732997894287 Total Reward: 563.6206896551626 Avg_Loss: 1.6033900325037853 Episode: 172: Time: 23.58040690422058 Total Reward: 541.6559485530476 Avg_Loss: 1.5898715816125149 Episode: 173: Time: 23.343698024749756 Total Reward: 628.0215827338044 Avg_Loss: 1.5734293728816409 Episode: 174: Time: 23.413686990737915 Total Reward: 584.310344827574 Avg_Loss: 1.6047735572362147 Episode: 175: Time: 23.481439352035522 Total Reward: 410.0505050504968 Avg_Loss: 1.6970932684024842 Episode: 176: Time: 23.36220073699951 Total Reward: 419.7058823529366 Avg_Loss: 1.5985108195733624 Episode: 177: Time: 23.3300838470459 Total Reward: 457.816901408446 Avg_Loss: 1.6031874328100382 Episode: 178: Time: 23.41885805130005 Total Reward: 281.94704049844415 Avg_Loss: 1.6301257282245059 Episode: 179: Time: 23.458149909973145 Total Reward: 217.30283911672382 Avg_Loss: 1.6178220019621008 Episode: 180: Time: 23.40424394607544 Total Reward: 499.7712418300574 Avg_Loss: 1.5951635945243996 Episode: 181: Time: 23.426961183547974 Total Reward: 680.4385964912163 Avg_Loss: 1.6269327046490516 Episode: 182: Time: 23.212927103042603 Total Reward: 579.1573033707775 Avg_Loss: 1.5107966631400485 Episode: 183: Time: 23.39946413040161 Total Reward: 459.794520547942 Avg_Loss: 1.593826770532031 Episode: 184: Time: 23.385694980621338 Total Reward: 579.2671009771892 Avg_Loss: 1.6911180610416316 Episode: 185: Time: 23.36878204345703 Total Reward: 494.65517241378416 Avg_Loss: 1.759625305147732 Episode: 186: Time: 23.311036109924316 Total Reward: 645.3508771929688 Avg_Loss: 1.6265691228273536 Episode: 187: Time: 23.457499980926514 Total Reward: 279.60317460317754 Avg_Loss: 1.5994046781243396 Episode: 188: Time: 23.262563228607178 Total Reward: 685.3921568627334 Avg_Loss: 1.6146119124248248 Episode: 189: Time: 23.282949209213257 Total Reward: 742.0370370370248 Avg_Loss: 1.497770161438389 Episode: 190: Time: 23.379179000854492 Total Reward: 416.86440677964936 Avg_Loss: 1.7060111961444886 Episode: 191: Time: 23.409074068069458 Total Reward: 567.068965517229 Avg_Loss: 1.696136780396229 Episode: 192: Time: 23.52620530128479 Total Reward: 501.59090909089826 Avg_Loss: 1.6117886189653092 Episode: 193: Time: 23.402414798736572 Total Reward: 663.0071174377126 Avg_Loss: 1.6931951947572852 Episode: 194: Time: 23.503334283828735 Total Reward: 545.7766990291154 Avg_Loss: 1.696703878771357 Episode: 195: Time: 23.285064935684204 Total Reward: 538.6996336996251 Avg_Loss: 1.5585565073650425 Episode: 196: Time: 23.626835107803345 Total Reward: 334.8507462686549 Avg_Loss: 1.6254877635911733 Episode: 197: Time: 23.619632959365845 Total Reward: 523.6186186186126 Avg_Loss: 1.6272733033705158 Episode: 198: Time: 23.444419145584106 Total Reward: 710.3691275167666 Avg_Loss: 1.5935558749347174 Episode: 199: Time: 23.321229934692383 Total Reward: 515.1083032490876 Avg_Loss: 1.617158796857385 Validation Mean Reward: 488.896060695705 Validation Std Reward: 195.2491675698184 Test Mean Reward: 494.10852604887606 Test Std Reward: 166.54038201578032
xxxxxxxxxxPlease include a plot of the training and validation rewards over the episodes in the report. An additional question to answer is does the loss matter in DQN? Why or why not?We can also draw a animation of the car in one game, the code is provided belowPlease include a plot of the training and validation rewards over the episodes in the report. An additional question to answer is does the loss matter in DQN? Why or why not?
We can also draw a animation of the car in one game, the code is provided below
xxxxxxxxxx**Ans.** Yes, the loss function matters. If the loss function is not decided correctly either the loss may decrease, the rewards won't increase over time necessarily. The choice of loss function in DQN is essential for training stability, convergence, and achieving optimal performance. Different loss functions may be suitable for different scenarios or variations of the DQN algorithm.To decide whether MSE loss is better or smooth L1 loss, we should pay attention to the charactristics of our problem. MSE loss tends to penalize larger errors more heavily, which can be suitable if we want to prioritize reducing large errors. However, it is more sensitive to outliers and can amplify their impact on the training process, potentially leading to slower convergence or instability; whereas Smooth L1 loss offers a combination of both L1 and L2 loss functions. It behaves like L2 loss for small errors and like L1 loss for large errors. Smooth L1 loss is less sensitive to outliers and can provide more robust training by reducing the impact of extreme errors. It can be particularly useful if the problem involves a high degree of noise or outliers in the target Q-values.As shown above, average loss is smaller for smooth L1 loss. The test mean rewards are nearly the same but standard deviation is smaller for smooth L1. For this case, smooth L1 worked better probably due to the existance of some noise.Ans. Yes, the loss function matters. If the loss function is not decided correctly either the loss may decrease, the rewards won't increase over time necessarily. The choice of loss function in DQN is essential for training stability, convergence, and achieving optimal performance. Different loss functions may be suitable for different scenarios or variations of the DQN algorithm.
To decide whether MSE loss is better or smooth L1 loss, we should pay attention to the charactristics of our problem. MSE loss tends to penalize larger errors more heavily, which can be suitable if we want to prioritize reducing large errors. However, it is more sensitive to outliers and can amplify their impact on the training process, potentially leading to slower convergence or instability; whereas Smooth L1 loss offers a combination of both L1 and L2 loss functions. It behaves like L2 loss for small errors and like L1 loss for large errors. Smooth L1 loss is less sensitive to outliers and can provide more robust training by reducing the impact of extreme errors. It can be particularly useful if the problem involves a high degree of noise or outliers in the target Q-values.
As shown above, average loss is smaller for smooth L1 loss. The test mean rewards are nearly the same but standard deviation is smaller for smooth L1. For this case, smooth L1 worked better probably due to the existance of some noise.
xxxxxxxxxx# Plots for a trainer trained by MSE lossimport matplotlib.pyplot as pltavg_losses, total_rewards, mean_test_rewards, std_test_rewards = out_mseplt.plot(total_rewards)plt.title("Total rewards")plt.show();plt.plot(avg_losses)plt.title("Average losses")plt.show();x = np.arange(50, 201, 50)plt.plot(x, mean_test_rewards)plt.title("Mean test rewards")plt.show();plt.plot(x, std_test_rewards)plt.title("Std test rewards")plt.show();xxxxxxxxxx# Plots for a trainer trained by smooth l1 lossavg_losses, total_rewards, mean_test_rewards, std_test_rewards = out_l1plt.plot(total_rewards)plt.title("Total rewards")plt.show();plt.plot(avg_losses)plt.title("Average losses")plt.show();x = np.arange(50, 201, 50)plt.plot(x, mean_test_rewards)plt.title("Mean test rewards")plt.show();plt.plot(x, std_test_rewards)plt.title("Std test rewards")plt.show();xxxxxxxxxx# animation for a trainer trained by MSE losseval_env = gym.make('CarRacing-v2', continuous=True, render_mode='rgb_array')eval_env = EnvWrapper(eval_env)total_rewards, frames = trainerDQN1.play_episode(0,True,42)anim = animate(frames)HTML(anim.to_jshtml())xxxxxxxxxx# animation for a trainer trained by smooth l1 losseval_env = gym.make('CarRacing-v2', continuous=True, render_mode='rgb_array')eval_env = EnvWrapper(eval_env)total_rewards, frames = trainerDQN2.play_episode(0,True,42)anim = animate(frames)HTML(anim.to_jshtml())xxxxxxxxxx### Double DQNIn the original paper, where the algorithim is shown above, the estimated target Q value was computed using the current Q network's weights. However, this can lead to overestimation of the Q values. To mitigate this, we can use the target network to compute the target Q value. This is known as Double DQN.#### Hard updating Target Network (5 points)Original implementations for this involved hard updates, where the model weights were copied to the target network every C steps. This is known as hard updating. This was what was used in the Nature Paper by Mnih et al 2015 "Human-level control through deep reinforcement learning"Please implement this by implementing the `_optimize_model` and `_update_model` classes in `HardUpdateDQN` in `DQN.py`.In the original paper, where the algorithim is shown above, the estimated target Q value was computed using the current Q network's weights. However, this can lead to overestimation of the Q values. To mitigate this, we can use the target network to compute the target Q value. This is known as Double DQN.
Original implementations for this involved hard updates, where the model weights were copied to the target network every C steps. This is known as hard updating. This was what was used in the Nature Paper by Mnih et al 2015 "Human-level control through deep reinforcement learning"
Please implement this by implementing the _optimize_model and _update_model classes in HardUpdateDQN in DQN.py.
xxxxxxxxxxtrainerHardUpdateDQN = DQN.HardUpdateDQN(EnvWrapper(env), model.Nature_Paper_Conv, update_freq = 100, lr = 0.00025, gamma = 0.95, buffer_size=100000, batch_size=32, loss_fn = "mse_loss", use_wandb = False, device = 'cpu', seed = 42, epsilon_scheduler = utils.exponential_decay(1, 1000,0.1), save_path = utils.get_save_path("DoubleDQN_HardUpdates/","./runs/"))trainerHardUpdateDQN.train(200,50,30,50,50)saving to ./runs/DoubleDQN_HardUpdates/run0 Episode: 1: Time: 17.792473793029785 Total Reward: -63.75000000000068 Avg_Loss: 0.630705767222176 Episode: 2: Time: 23.915672063827515 Total Reward: -84.16967509025231 Avg_Loss: 0.52350636603342 Episode: 3: Time: 24.071273803710938 Total Reward: -31.09022556391058 Avg_Loss: 0.5189976973044333 Episode: 4: Time: 23.95846199989319 Total Reward: -30.483870967742114 Avg_Loss: 0.610305790711163 Episode: 5: Time: 24.009126901626587 Total Reward: -7.33766233766292 Avg_Loss: 0.674683321549111 Episode: 6: Time: 24.103214025497437 Total Reward: -53.47750865051944 Avg_Loss: 0.7039703618609855 Episode: 7: Time: 24.01707911491394 Total Reward: -23.57142857142929 Avg_Loss: 0.7277495655379876 Episode: 8: Time: 23.977355003356934 Total Reward: -78.16498316498306 Avg_Loss: 0.6769561887006549 Episode: 9: Time: 24.818693161010742 Total Reward: -31.50793650793713 Avg_Loss: 0.6265656726763529 Episode: 10: Time: 24.082226991653442 Total Reward: -30.18518518518591 Avg_Loss: 0.6642914196131986 Episode: 11: Time: 24.045199155807495 Total Reward: -1.6666666666666028 Avg_Loss: 0.7320616529831866 Episode: 12: Time: 24.052443981170654 Total Reward: 44.93174061433691 Avg_Loss: 0.9139129687413698 Episode: 13: Time: 23.85068988800049 Total Reward: -24.328621908127904 Avg_Loss: 1.026351731834041 Episode: 14: Time: 24.945476055145264 Total Reward: 297.2261484098839 Avg_Loss: 1.2644980702893573 Episode: 15: Time: 24.489127159118652 Total Reward: 336.8936877076407 Avg_Loss: 1.5278667289168895 Episode: 16: Time: 24.25092124938965 Total Reward: 388.0188679245182 Avg_Loss: 1.8232483788698661 Episode: 17: Time: 24.319631814956665 Total Reward: 200.38461538461954 Avg_Loss: 2.0219053126433315 Episode: 18: Time: 24.00578784942627 Total Reward: 19.186851211073893 Avg_Loss: 2.0739796490979794 Episode: 19: Time: 11.576121091842651 Total Reward: -95.50000000000037 Avg_Loss: 2.1165331719737304 Episode: 20: Time: 24.242506980895996 Total Reward: -0.7239057239054203 Avg_Loss: 3.184624611705291 Episode: 21: Time: 24.133284091949463 Total Reward: 339.30656934306404 Avg_Loss: 4.715633521310422 Episode: 22: Time: 24.093611001968384 Total Reward: 145.54982817869842 Avg_Loss: 3.873302436676346 Episode: 23: Time: 24.137619018554688 Total Reward: 203.18181818182248 Avg_Loss: 5.189784720039167 Episode: 24: Time: 24.17206382751465 Total Reward: 132.42474916388386 Avg_Loss: 2.932057719771602 Episode: 25: Time: 21.907738208770752 Total Reward: -126.26655290102451 Avg_Loss: 4.435361351208253 Episode: 26: Time: 24.59030318260193 Total Reward: 279.9999999999998 Avg_Loss: 6.892269643665362 Episode: 27: Time: 24.19022274017334 Total Reward: 21.60777385159183 Avg_Loss: 8.870944892658907 Episode: 28: Time: 26.345628023147583 Total Reward: 136.31672597865196 Avg_Loss: 7.0525425457153 Episode: 29: Time: 24.767261028289795 Total Reward: 548.3121019108149 Avg_Loss: 3.4879929215467276 Episode: 30: Time: 24.317114114761353 Total Reward: 132.27272727272907 Avg_Loss: 6.0149259444545295 Episode: 31: Time: 24.477201223373413 Total Reward: 157.78810408922382 Avg_Loss: 4.515603584526968 Episode: 32: Time: 25.00206208229065 Total Reward: 248.0232558139566 Avg_Loss: 5.633008697203228 Episode: 33: Time: 24.425124883651733 Total Reward: 391.92810457515355 Avg_Loss: 4.871360142942236 Episode: 34: Time: 24.57722282409668 Total Reward: 424.9999999999883 Avg_Loss: 5.144679617981951 Episode: 35: Time: 24.259000778198242 Total Reward: 404.9999999999905 Avg_Loss: 6.706640308394151 Episode: 36: Time: 24.19731879234314 Total Reward: 555.9090909090772 Avg_Loss: 5.6151600835704 Episode: 37: Time: 24.396366119384766 Total Reward: 304.36102236421067 Avg_Loss: 6.632896581617724 Episode: 38: Time: 24.94207787513733 Total Reward: 412.09219858154944 Avg_Loss: 6.415923565876584 Episode: 39: Time: 25.664610147476196 Total Reward: 312.1661237784973 Avg_Loss: 6.382485403734095 Episode: 40: Time: 24.68681287765503 Total Reward: 538.0645161290277 Avg_Loss: 6.133016110969191 Episode: 41: Time: 24.27351999282837 Total Reward: 476.9298245613927 Avg_Loss: 6.778691347907571 Episode: 42: Time: 24.704457998275757 Total Reward: 811.0150375939709 Avg_Loss: 7.2234457771317295 Episode: 43: Time: 23.940546989440918 Total Reward: 626.5686274509742 Avg_Loss: 6.590719760466023 Episode: 44: Time: 24.08778190612793 Total Reward: 731.6666666666557 Avg_Loss: 6.030666870730264 Episode: 45: Time: 24.152155876159668 Total Reward: 479.39446366780885 Avg_Loss: 6.587396972820539 Episode: 46: Time: 24.259055852890015 Total Reward: 452.8927203065037 Avg_Loss: 7.979168384015059 Episode: 47: Time: 24.480200052261353 Total Reward: 569.473684210514 Avg_Loss: 7.880866835097305 Episode: 48: Time: 24.227317094802856 Total Reward: 338.56643356642434 Avg_Loss: 7.856146791902911 Episode: 49: Time: 24.384070873260498 Total Reward: 438.98058252426546 Avg_Loss: 7.173105302978964 Validation Mean Reward: 158.48891104886124 Validation Std Reward: 312.49261802619657 Episode: 50: Time: 23.924965858459473 Total Reward: 362.04467353950946 Avg_Loss: 7.880121706914501 Episode: 51: Time: 24.18545889854431 Total Reward: -49.545454545454945 Avg_Loss: 7.793948990958078 Episode: 52: Time: 24.362265825271606 Total Reward: 477.91666666666094 Avg_Loss: 7.856699665554431 Episode: 53: Time: 24.12750005722046 Total Reward: 366.7940199335485 Avg_Loss: 9.103180164048652 Episode: 54: Time: 24.014076232910156 Total Reward: -31.842105263158597 Avg_Loss: 8.828528865044857 Episode: 55: Time: 24.005878925323486 Total Reward: 467.2895622895547 Avg_Loss: 8.536039561283689 Episode: 56: Time: 24.18026113510132 Total Reward: 460.2238805970088 Avg_Loss: 9.175093460483712 Episode: 57: Time: 24.22234010696411 Total Reward: 489.37499999999227 Avg_Loss: 9.472669286888186 Episode: 58: Time: 25.207301139831543 Total Reward: 529.9999999999902 Avg_Loss: 9.555317507571534 Episode: 59: Time: 24.800809860229492 Total Reward: 383.54785478547683 Avg_Loss: 8.28663476274795 Episode: 60: Time: 24.812219858169556 Total Reward: 475.85020242914214 Avg_Loss: 7.844767563984174 Episode: 61: Time: 24.37883996963501 Total Reward: 334.65779467680716 Avg_Loss: 7.6952012887521954 Episode: 62: Time: 24.632569074630737 Total Reward: 310.0632911392357 Avg_Loss: 8.958022881956662 Episode: 63: Time: 24.574854850769043 Total Reward: 452.7707006369322 Avg_Loss: 9.103310517904138 Episode: 64: Time: 24.477356910705566 Total Reward: 380.8620689655087 Avg_Loss: 9.465872565738293 Episode: 65: Time: 23.907142162322998 Total Reward: 599.6564885496077 Avg_Loss: 9.309627862537608 Episode: 66: Time: 24.60072374343872 Total Reward: 483.39721254354606 Avg_Loss: 8.866608706842952 Episode: 67: Time: 24.271952867507935 Total Reward: 328.333333333329 Avg_Loss: 8.058929167875723 Episode: 68: Time: 24.735481023788452 Total Reward: 273.2432432432464 Avg_Loss: 7.05420832323427 Episode: 69: Time: 24.073646783828735 Total Reward: 264.5890410958941 Avg_Loss: 7.415294947744417 Episode: 70: Time: 24.15129017829895 Total Reward: 225.51282051282485 Avg_Loss: 7.905773579072552 Episode: 71: Time: 24.087978839874268 Total Reward: 599.3396226414986 Avg_Loss: 7.486186279970057 Episode: 72: Time: 24.500232934951782 Total Reward: 449.8717948717881 Avg_Loss: 7.285564244795246 Episode: 73: Time: 24.032716035842896 Total Reward: 518.8996138996015 Avg_Loss: 7.135146640929856 Episode: 74: Time: 24.25556993484497 Total Reward: 321.3701067615635 Avg_Loss: 7.566306080136981 Episode: 75: Time: 24.521398067474365 Total Reward: 425.59925093632296 Avg_Loss: 7.807855183336915 Episode: 76: Time: 24.614328861236572 Total Reward: 397.06349206348824 Avg_Loss: 7.782733762965483 Episode: 77: Time: 24.063436031341553 Total Reward: 518.7184115523357 Avg_Loss: 7.844762186543281 Episode: 78: Time: 24.587759017944336 Total Reward: 365.2076124567467 Avg_Loss: 7.886296798201168 Episode: 79: Time: 23.88755488395691 Total Reward: 543.989169675079 Avg_Loss: 7.848050462097681 Episode: 80: Time: 24.054147958755493 Total Reward: 553.5507246376749 Avg_Loss: 7.674668889085786 Episode: 81: Time: 24.500955820083618 Total Reward: 382.7070063694179 Avg_Loss: 7.56750219709733 Episode: 82: Time: 24.304293870925903 Total Reward: 456.49501661128534 Avg_Loss: 7.536523745841339 Episode: 83: Time: 24.12336492538452 Total Reward: 376.5189873417649 Avg_Loss: 7.969628735750663
Episode: 84: Time: 24.211642026901245 Total Reward: 567.1160409556212 Avg_Loss: 8.189190772401185 Episode: 85: Time: 19.145009756088257 Total Reward: 394.5833333333301 Avg_Loss: 7.5477354035658 Episode: 86: Time: 18.706461906433105 Total Reward: 455.5226480836171 Avg_Loss: 7.674428782042335 Episode: 87: Time: 18.69327712059021 Total Reward: 605.3484320557403 Avg_Loss: 7.52655442923057 Episode: 88: Time: 18.475342273712158 Total Reward: 431.7175572518991 Avg_Loss: 8.3595830242173 Episode: 89: Time: 18.417510986328125 Total Reward: 238.33333333333516 Avg_Loss: 8.323326017676282 Episode: 90: Time: 18.507752895355225 Total Reward: 273.2432432432443 Avg_Loss: 8.590620316377207 Episode: 91: Time: 18.715154886245728 Total Reward: 150.73378839590598 Avg_Loss: 8.675868487157741 Episode: 92: Time: 18.553852796554565 Total Reward: 329.05063291139226 Avg_Loss: 8.636651369703918 Episode: 93: Time: 18.37106990814209 Total Reward: 301.8871595330669 Avg_Loss: 8.163391015108894 Episode: 94: Time: 18.918802976608276 Total Reward: 209.09356725146392 Avg_Loss: 8.810710800795995 Episode: 95: Time: 18.785063982009888 Total Reward: 319.38356164382867 Avg_Loss: 7.425076534267233 Episode: 96: Time: 18.525031089782715 Total Reward: 366.8181818181788 Avg_Loss: 7.51912088103655 Episode: 97: Time: 18.495784997940063 Total Reward: 278.4939759036168 Avg_Loss: 6.705324340768221 Episode: 98: Time: 18.705090761184692 Total Reward: 172.65799256505798 Avg_Loss: 7.36710786819458 Episode: 99: Time: 19.473571062088013 Total Reward: 293.3161512027475 Avg_Loss: 7.820572058693702 Validation Mean Reward: 381.60434067184906 Validation Std Reward: 141.3323517759644 Episode: 100: Time: 19.16981601715088 Total Reward: 347.5675675675638 Avg_Loss: 6.920303670298152 Episode: 101: Time: 19.24945616722107 Total Reward: 465.43956043955586 Avg_Loss: 6.852322181232837 Episode: 102: Time: 19.007564783096313 Total Reward: 274.3379790940766 Avg_Loss: 7.548333150999887 Episode: 103: Time: 19.1750431060791 Total Reward: 354.40476190475727 Avg_Loss: 6.7337053523344155 Episode: 104: Time: 18.761635780334473 Total Reward: 259.3859649122836 Avg_Loss: 7.015599027401259 Episode: 105: Time: 18.884927988052368 Total Reward: 313.94568690095247 Avg_Loss: 7.984504037544507 Episode: 106: Time: 18.879838943481445 Total Reward: 97.8571428571465 Avg_Loss: 7.562381426827247 Episode: 107: Time: 18.835592031478882 Total Reward: 391.5771812080479 Avg_Loss: 7.527927171282408 Episode: 108: Time: 18.939391136169434 Total Reward: 300.42483660130625 Avg_Loss: 8.03086661190546 Episode: 109: Time: 18.76619029045105 Total Reward: 361.9288389513086 Avg_Loss: 8.243176022497545 Episode: 110: Time: 18.651581048965454 Total Reward: 386.3559322033883 Avg_Loss: 7.717727811396623 Episode: 111: Time: 18.802273988723755 Total Reward: 362.337883959042 Avg_Loss: 7.311729770748555 Episode: 112: Time: 18.872798204421997 Total Reward: 298.25842696629365 Avg_Loss: 7.505096206144125 Episode: 113: Time: 18.421224355697632 Total Reward: 335.2325581395363 Avg_Loss: 7.342528465415249 Episode: 114: Time: 18.583197832107544 Total Reward: 269.5161290322611 Avg_Loss: 7.437377448843307 Episode: 115: Time: 18.59877586364746 Total Reward: 469.10256410255624 Avg_Loss: 7.607444764185352 Episode: 116: Time: 18.302746057510376 Total Reward: 145.7407407407442 Avg_Loss: 8.022527585510446 Episode: 117: Time: 18.620620250701904 Total Reward: 238.333333333337 Avg_Loss: 8.075675878204217 Episode: 118: Time: 18.566111087799072 Total Reward: 337.33082706766857 Avg_Loss: 6.57846649754949 Episode: 119: Time: 18.612130880355835 Total Reward: 321.66666666666526 Avg_Loss: 7.431139280816086 Episode: 120: Time: 18.603684902191162 Total Reward: 179.50980392157166 Avg_Loss: 7.714402364081695 Episode: 121: Time: 18.918317079544067 Total Reward: 262.3770491803314 Avg_Loss: 8.339316675642959 Episode: 122: Time: 18.932281970977783 Total Reward: 398.10344827585516 Avg_Loss: 7.477603585780168 Episode: 123: Time: 18.955049991607666 Total Reward: 219.1993957703968 Avg_Loss: 7.850806267321611 Episode: 124: Time: 18.53740906715393 Total Reward: 367.68656716417274 Avg_Loss: 6.970319176922326 Episode: 125: Time: 18.466306924819946 Total Reward: 366.0169491525399 Avg_Loss: 7.646037825015413 Episode: 126: Time: 19.553996086120605 Total Reward: 48.34470989761012 Avg_Loss: 8.016427755355835 Episode: 127: Time: 19.404576063156128 Total Reward: 431.31578947367393 Avg_Loss: 8.603155392558635 Episode: 128: Time: 19.045630931854248 Total Reward: 410.2264808362343 Avg_Loss: 6.865586748143204 Episode: 129: Time: 18.70679020881653 Total Reward: 343.2352941176451 Avg_Loss: 7.803348468131378 Episode: 130: Time: 18.585413932800293 Total Reward: 440.9999999999962 Avg_Loss: 7.657811242993138 Episode: 131: Time: 18.659744024276733 Total Reward: 388.22147651006566 Avg_Loss: 8.064709050314766 Episode: 132: Time: 18.813589096069336 Total Reward: 289.39306358381555 Avg_Loss: 7.903340857069032 Episode: 133: Time: 18.580821990966797 Total Reward: 328.5668789808836 Avg_Loss: 8.118648772480107 Episode: 134: Time: 18.56669807434082 Total Reward: 376.6981132075453 Avg_Loss: 8.23798922630919 Episode: 135: Time: 18.732296228408813 Total Reward: 379.90347490346346 Avg_Loss: 9.17326431715188 Episode: 136: Time: 20.173675775527954 Total Reward: 208.2069970845513 Avg_Loss: 8.626357399115042 Episode: 137: Time: 19.411225080490112 Total Reward: 468.57388316150286 Avg_Loss: 9.174216849463326 Episode: 138: Time: 18.616836071014404 Total Reward: 290.454545454543 Avg_Loss: 8.845614285028281 Episode: 139: Time: 18.728473901748657 Total Reward: 470.3846153846071 Avg_Loss: 7.817431569099426 Episode: 140: Time: 18.680349111557007 Total Reward: 464.5667870035996 Avg_Loss: 7.758364448026449 Episode: 141: Time: 18.762307167053223 Total Reward: 657.7675276752672 Avg_Loss: 7.762972212639175 Episode: 142: Time: 19.408571243286133 Total Reward: 414.37499999999307 Avg_Loss: 7.8318701782146425 Episode: 143: Time: 19.574238061904907 Total Reward: 274.28104575163434 Avg_Loss: 7.818371706650037 Episode: 144: Time: 18.818320989608765 Total Reward: 443.7323943661914 Avg_Loss: 8.255099798450951 Episode: 145: Time: 18.723212003707886 Total Reward: 391.5771812080515 Avg_Loss: 7.954092494579924 Episode: 146: Time: 19.294965982437134 Total Reward: 439.0501792114639 Avg_Loss: 7.791874443783479 Episode: 147: Time: 19.06006407737732 Total Reward: 591.3468634686267 Avg_Loss: 7.80911895507524 Episode: 148: Time: 19.27340817451477 Total Reward: 434.6296296296209 Avg_Loss: 7.1285854167297105 Episode: 149: Time: 18.74958634376526 Total Reward: 423.38235294117237 Avg_Loss: 7.6656743969236105 Validation Mean Reward: 570.829705120681 Validation Std Reward: 165.5552750920016 Episode: 150: Time: 18.764787912368774 Total Reward: 352.18309859154886 Avg_Loss: 7.203345960929614 Episode: 151: Time: 18.760167121887207 Total Reward: 384.87616099070954 Avg_Loss: 7.4530276651142024 Episode: 152: Time: 18.75018286705017 Total Reward: -39.827586206897266 Avg_Loss: 7.18591606516798 Episode: 153: Time: 18.951334714889526 Total Reward: 490.8585858585781 Avg_Loss: 7.23057036139384 Episode: 154: Time: 18.758795022964478 Total Reward: 606.8072289156528 Avg_Loss: 7.427789193241536 Episode: 155: Time: 18.619415044784546 Total Reward: 457.34657039710686 Avg_Loss: 8.111380177385668 Episode: 156: Time: 18.549362897872925 Total Reward: 582.3049645390016 Avg_Loss: 7.949010664675416 Episode: 157: Time: 19.23068618774414 Total Reward: 437.89473684209645 Avg_Loss: 7.83748080249594 Episode: 158: Time: 18.897192001342773 Total Reward: 455.17301038061873 Avg_Loss: 7.112979190690177 Episode: 159: Time: 18.96295976638794 Total Reward: 616.6104868913774 Avg_Loss: 7.846901720311461 Episode: 160: Time: 19.169079780578613 Total Reward: 346.8604651162694 Avg_Loss: 7.7942222897745985 Episode: 161: Time: 19.019866943359375 Total Reward: 413.03858520899655 Avg_Loss: 8.15929454014081 Episode: 162: Time: 19.0905818939209 Total Reward: 631.9503546099237 Avg_Loss: 8.550050957864071 Episode: 163: Time: 19.651039123535156 Total Reward: 592.4999999999917 Avg_Loss: 7.961214769788149 Episode: 164: Time: 18.92538070678711 Total Reward: 579.8251748251696 Avg_Loss: 7.936232039908401 Episode: 165: Time: 19.566930055618286 Total Reward: 651.7105263157758 Avg_Loss: 7.324912637221713
Episode: 166: Time: 17.522963285446167 Total Reward: 243.59368770763297 Avg_Loss: 7.218133349224453 Episode: 167: Time: 18.80939483642578 Total Reward: 619.2857142857035 Avg_Loss: 7.901547805601809 Episode: 168: Time: 18.587332010269165 Total Reward: 500.46925566342145 Avg_Loss: 8.134706851814975 Episode: 169: Time: 18.81227684020996 Total Reward: 442.73584905659885 Avg_Loss: 7.920106151524712 Episode: 170: Time: 19.098884105682373 Total Reward: 353.97959183673294 Avg_Loss: 7.982888856855761 Episode: 171: Time: 18.691712141036987 Total Reward: 595.9090909090784 Avg_Loss: 7.2928816741254145 Episode: 172: Time: 18.66236400604248 Total Reward: 575.2508960573359 Avg_Loss: 8.388004270922236 Episode: 173: Time: 18.818603038787842 Total Reward: 545.522875816986 Avg_Loss: 8.549828620517955 Episode: 174: Time: 18.928699254989624 Total Reward: 456.85185185183957 Avg_Loss: 7.820556531433298 Episode: 175: Time: 18.62218976020813 Total Reward: 481.47058823528437 Avg_Loss: 7.87028297256021 Episode: 176: Time: 18.79270100593567 Total Reward: 443.4615384615304 Avg_Loss: 8.157694604216504 Episode: 177: Time: 18.973999738693237 Total Reward: 536.9444444444343 Avg_Loss: 7.972652728818044 Episode: 178: Time: 18.767948150634766 Total Reward: 487.4175824175762 Avg_Loss: 8.547904504447423 Episode: 179: Time: 18.687373876571655 Total Reward: 582.7408637873668 Avg_Loss: 8.136691423023448 Episode: 180: Time: 16.044176816940308 Total Reward: 65.55882352941498 Avg_Loss: 8.631039306229235 Episode: 181: Time: 18.633880853652954 Total Reward: 320.4929577464793 Avg_Loss: 8.13483497976255 Episode: 182: Time: 18.706177711486816 Total Reward: 166.68224299065702 Avg_Loss: 7.65157396853471 Episode: 183: Time: 18.84642505645752 Total Reward: 484.57957957957575 Avg_Loss: 7.349135001166528 Episode: 184: Time: 19.705039262771606 Total Reward: 450.77464788731345 Avg_Loss: 8.723133633116714 Episode: 185: Time: 19.052767992019653 Total Reward: 485.3278688524505 Avg_Loss: 8.295316724216237 Episode: 186: Time: 18.747542142868042 Total Reward: 577.5978647686734 Avg_Loss: 9.24712409091597 Episode: 187: Time: 18.808973789215088 Total Reward: 611.9597069596994 Avg_Loss: 9.258875172679163 Episode: 188: Time: 19.355461835861206 Total Reward: 581.2589928057444 Avg_Loss: 8.928479510195116 Episode: 189: Time: 19.070024967193604 Total Reward: 546.5094339622518 Avg_Loss: 9.52313501794799 Episode: 190: Time: 19.290614128112793 Total Reward: 453.6111111111059 Avg_Loss: 9.216579639611124 Episode: 191: Time: 19.12901782989502 Total Reward: 587.9268292682832 Avg_Loss: 8.46572141887761 Episode: 192: Time: 19.16946315765381 Total Reward: 466.53846153845427 Avg_Loss: 8.770868108052166 Episode: 193: Time: 19.348088264465332 Total Reward: 473.8405797101385 Avg_Loss: 8.152582584308977 Episode: 194: Time: 18.878765106201172 Total Reward: 522.7474402730265 Avg_Loss: 8.045219347256573 Episode: 195: Time: 19.625776052474976 Total Reward: 346.7344173441687 Avg_Loss: 7.973953552606727 Episode: 196: Time: 18.920156240463257 Total Reward: 332.6315789473683 Avg_Loss: 8.619011034484672 Episode: 197: Time: 19.082797050476074 Total Reward: 564.4202898550662 Avg_Loss: 8.563340574753385 Episode: 198: Time: 19.62026810646057 Total Reward: 305.64102564101677 Avg_Loss: 8.67755934370666 Episode: 199: Time: 19.23921298980713 Total Reward: 400.4954954954925 Avg_Loss: 8.053310499471777 Validation Mean Reward: 373.38259773815156 Validation Std Reward: 220.00550470225144 Test Mean Reward: 486.0698676629879 Test Std Reward: 265.9200378661544
([0.630705767222176, 0.52350636603342, 0.5189976973044333, 0.610305790711163, 0.674683321549111, 0.7039703618609855, 0.7277495655379876, 0.6769561887006549, 0.6265656726763529, 0.6642914196131986, 0.7320616529831866, 0.9139129687413698, 1.026351731834041, 1.2644980702893573, 1.5278667289168895, 1.8232483788698661, 2.0219053126433315, 2.0739796490979794, 2.1165331719737304, 3.184624611705291, 4.715633521310422, 3.873302436676346, 5.189784720039167, 2.932057719771602, 4.435361351208253, 6.892269643665362, 8.870944892658907, 7.0525425457153, 3.4879929215467276, 6.0149259444545295, 4.515603584526968, 5.633008697203228, 4.871360142942236, 5.144679617981951, 6.706640308394151, 5.6151600835704, 6.632896581617724, 6.415923565876584, 6.382485403734095, 6.133016110969191, 6.778691347907571, 7.2234457771317295, 6.590719760466023, 6.030666870730264, 6.587396972820539, 7.979168384015059, 7.880866835097305, 7.856146791902911, 7.173105302978964, 7.880121706914501, 7.793948990958078, 7.856699665554431, 9.103180164048652, 8.828528865044857, 8.536039561283689, 9.175093460483712, 9.472669286888186, 9.555317507571534, 8.28663476274795, 7.844767563984174, 7.6952012887521954, 8.958022881956662, 9.103310517904138, 9.465872565738293, 9.309627862537608, 8.866608706842952, 8.058929167875723, 7.05420832323427, 7.415294947744417, 7.905773579072552, 7.486186279970057, 7.285564244795246, 7.135146640929856, 7.566306080136981, 7.807855183336915, 7.782733762965483, 7.844762186543281, 7.886296798201168, 7.848050462097681, 7.674668889085786, 7.56750219709733, 7.536523745841339, 7.969628735750663, 8.189190772401185, 7.5477354035658, 7.674428782042335, 7.52655442923057, 8.3595830242173, 8.323326017676282, 8.590620316377207, 8.675868487157741, 8.636651369703918, 8.163391015108894, 8.810710800795995, 7.425076534267233, 7.51912088103655, 6.705324340768221, 7.36710786819458, 7.820572058693702, 6.920303670298152, 6.852322181232837, 7.548333150999887, 6.7337053523344155, 7.015599027401259, 7.984504037544507, 7.562381426827247, 7.527927171282408, 8.03086661190546, 8.243176022497545, 7.717727811396623, 7.311729770748555, 7.505096206144125, 7.342528465415249, 7.437377448843307, 7.607444764185352, 8.022527585510446, 8.075675878204217, 6.57846649754949, 7.431139280816086, 7.714402364081695, 8.339316675642959, 7.477603585780168, 7.850806267321611, 6.970319176922326, 7.646037825015413, 8.016427755355835, 8.603155392558635, 6.865586748143204, 7.803348468131378, 7.657811242993138, 8.064709050314766, 7.903340857069032, 8.118648772480107, 8.23798922630919, 9.17326431715188, 8.626357399115042, 9.174216849463326, 8.845614285028281, 7.817431569099426, 7.758364448026449, 7.762972212639175, 7.8318701782146425, 7.818371706650037, 8.255099798450951, 7.954092494579924, 7.791874443783479, 7.80911895507524, 7.1285854167297105, 7.6656743969236105, 7.203345960929614, 7.4530276651142024, 7.18591606516798, 7.23057036139384, 7.427789193241536, 8.111380177385668, 7.949010664675416, 7.83748080249594, 7.112979190690177, 7.846901720311461, 7.7942222897745985, 8.15929454014081, 8.550050957864071, 7.961214769788149, 7.936232039908401, 7.324912637221713, 7.218133349224453, 7.901547805601809, 8.134706851814975, 7.920106151524712, 7.982888856855761, 7.2928816741254145, 8.388004270922236, 8.549828620517955, 7.820556531433298, 7.87028297256021, 8.157694604216504, 7.972652728818044, 8.547904504447423, 8.136691423023448, 8.631039306229235, 8.13483497976255, 7.65157396853471, 7.349135001166528, 8.723133633116714, 8.295316724216237, 9.24712409091597, 9.258875172679163, 8.928479510195116, 9.52313501794799, 9.216579639611124, 8.46572141887761, 8.770868108052166, 8.152582584308977, 8.045219347256573, 7.973953552606727, 8.619011034484672, 8.563340574753385, 8.67755934370666, 8.053310499471777], [-63.75000000000068, -84.16967509025231, -31.09022556391058, -30.483870967742114, -7.33766233766292, -53.47750865051944, -23.57142857142929, -78.16498316498306, -31.50793650793713, -30.18518518518591, -1.6666666666666028, 44.93174061433691, -24.328621908127904, 297.2261484098839, 336.8936877076407, 388.0188679245182, 200.38461538461954, 19.186851211073893, -95.50000000000037, -0.7239057239054203, 339.30656934306404, 145.54982817869842, 203.18181818182248, 132.42474916388386, -126.26655290102451, 279.9999999999998, 21.60777385159183, 136.31672597865196, 548.3121019108149, 132.27272727272907, 157.78810408922382, 248.0232558139566, 391.92810457515355, 424.9999999999883, 404.9999999999905, 555.9090909090772, 304.36102236421067, 412.09219858154944, 312.1661237784973, 538.0645161290277, 476.9298245613927, 811.0150375939709, 626.5686274509742, 731.6666666666557, 479.39446366780885, 452.8927203065037, 569.473684210514, 338.56643356642434, 438.98058252426546, 362.04467353950946, -49.545454545454945, 477.91666666666094, 366.7940199335485, -31.842105263158597, 467.2895622895547, 460.2238805970088, 489.37499999999227, 529.9999999999902, 383.54785478547683, 475.85020242914214, 334.65779467680716, 310.0632911392357, 452.7707006369322, 380.8620689655087, 599.6564885496077, 483.39721254354606, 328.333333333329, 273.2432432432464, 264.5890410958941, 225.51282051282485, 599.3396226414986, 449.8717948717881, 518.8996138996015, 321.3701067615635, 425.59925093632296, 397.06349206348824, 518.7184115523357, 365.2076124567467, 543.989169675079, 553.5507246376749, 382.7070063694179, 456.49501661128534, 376.5189873417649, 567.1160409556212, 394.5833333333301, 455.5226480836171, 605.3484320557403, 431.7175572518991, 238.33333333333516, 273.2432432432443, 150.73378839590598, 329.05063291139226, 301.8871595330669, 209.09356725146392, 319.38356164382867, 366.8181818181788, 278.4939759036168, 172.65799256505798, 293.3161512027475, 347.5675675675638, 465.43956043955586, 274.3379790940766, 354.40476190475727, 259.3859649122836, 313.94568690095247, 97.8571428571465, 391.5771812080479, 300.42483660130625, 361.9288389513086, 386.3559322033883, 362.337883959042, 298.25842696629365, 335.2325581395363, 269.5161290322611, 469.10256410255624, 145.7407407407442, 238.333333333337, 337.33082706766857, 321.66666666666526, 179.50980392157166, 262.3770491803314, 398.10344827585516, 219.1993957703968, 367.68656716417274, 366.0169491525399, 48.34470989761012, 431.31578947367393, 410.2264808362343, 343.2352941176451, 440.9999999999962, 388.22147651006566, 289.39306358381555, 328.5668789808836, 376.6981132075453, 379.90347490346346, 208.2069970845513, 468.57388316150286, 290.454545454543, 470.3846153846071, 464.5667870035996, 657.7675276752672, 414.37499999999307, 274.28104575163434, 443.7323943661914, 391.5771812080515, 439.0501792114639, 591.3468634686267, 434.6296296296209, 423.38235294117237, 352.18309859154886, 384.87616099070954, -39.827586206897266, 490.8585858585781, 606.8072289156528, 457.34657039710686, 582.3049645390016, 437.89473684209645, 455.17301038061873, 616.6104868913774, 346.8604651162694, 413.03858520899655, 631.9503546099237, 592.4999999999917, 579.8251748251696, 651.7105263157758, 243.59368770763297, 619.2857142857035, 500.46925566342145, 442.73584905659885, 353.97959183673294, 595.9090909090784, 575.2508960573359, 545.522875816986, 456.85185185183957, 481.47058823528437, 443.4615384615304, 536.9444444444343, 487.4175824175762, 582.7408637873668, 65.55882352941498, 320.4929577464793, 166.68224299065702, 484.57957957957575, 450.77464788731345, 485.3278688524505, 577.5978647686734, 611.9597069596994, 581.2589928057444, 546.5094339622518, 453.6111111111059, 587.9268292682832, 466.53846153845427, 473.8405797101385, 522.7474402730265, 346.7344173441687, 332.6315789473683, 564.4202898550662, 305.64102564101677, 400.4954954954925], [158.48891104886124, 381.60434067184906, 570.829705120681, 373.38259773815156], [312.49261802619657, 141.3323517759644, 165.5552750920016, 220.00550470225144])
xxxxxxxxxxtotal_rewards, frames = trainerHardUpdateDQN.play_episode(0,True,42)anim = animate(frames)HTML(anim.to_jshtml())xxxxxxxxxx#### Soft Updates (5 points)A more recent improvement is to use soft updates, also known as Polyak averaging, where the target network is updated with a small fraction of the current model weights every step. In other words:$$\theta_{target} = \tau \theta_{model} + (1-\tau) \theta_{target}$$for some $\tau << 1$Please implement this by implementing the `_update_model` class in `SoftUpdateDQN` in `DQN.py`.A more recent improvement is to use soft updates, also known as Polyak averaging, where the target network is updated with a small fraction of the current model weights every step. In other words:
_update_model class in SoftUpdateDQN in DQN.py.
xxxxxxxxxxtraineSoftUpdateDQN = DQN.SoftUpdateDQN(EnvWrapper(env), model.Nature_Paper_Conv, tau = 0.01, update_freq = 1, lr = 0.00025, gamma = 0.95, buffer_size=100000, batch_size=32, loss_fn = "mse_loss", use_wandb = False, device = 'cpu', seed = 42, epsilon_scheduler = utils.exponential_decay(1, 1000,0.1), save_path = utils.get_save_path("DoubleDQN_SoftUpdates","./runs/"))traineSoftUpdateDQN.train(200,50,30,50,50)saving to ./runs/DoubleDQN_SoftUpdates/run0 Episode: 1: Time: 13.38784122467041 Total Reward: -44.64028776978459 Avg_Loss: 0.7032929947659089 Episode: 2: Time: 18.21323871612549 Total Reward: -56.0389610389619 Avg_Loss: 0.6477146975758696 Episode: 3: Time: 18.280731201171875 Total Reward: -59.78873239436666 Avg_Loss: 0.5800422116832322 Episode: 4: Time: 18.35274910926819 Total Reward: -5.394265232975001 Avg_Loss: 0.6849102419370613 Episode: 5: Time: 19.05089044570923 Total Reward: -32.72893772893818 Avg_Loss: 0.7501259394177869 Episode: 6: Time: 18.543907165527344 Total Reward: -6.184210526316598 Avg_Loss: 0.7486689495658424 Episode: 7: Time: 18.786283016204834 Total Reward: 36.94444444444324 Avg_Loss: 0.8814797621815145 Episode: 8: Time: 22.17089295387268 Total Reward: -33.628158844765835 Avg_Loss: 0.9176623095609561 Episode: 9: Time: 24.634217977523804 Total Reward: -12.293233082707298 Avg_Loss: 0.871715443678388 Episode: 10: Time: 24.014771223068237 Total Reward: -26.8996415770615 Avg_Loss: 0.9815360074784575 Episode: 11: Time: 24.2324800491333 Total Reward: -36.55844155844217 Avg_Loss: 1.0410380279567062 Episode: 12: Time: 24.323740243911743 Total Reward: -18.87543252595222 Avg_Loss: 1.1218535047368843 Episode: 13: Time: 24.46326780319214 Total Reward: -34.28571428571476 Avg_Loss: 1.2063556467848164 Episode: 14: Time: 22.902597904205322 Total Reward: -150.19595959596012 Avg_Loss: 1.27918864619233 Episode: 15: Time: 23.485297918319702 Total Reward: -139.13174603174642 Avg_Loss: 4.027322133627316 Episode: 16: Time: 23.971450805664062 Total Reward: -42.530864197531216 Avg_Loss: 4.554474374764607 Episode: 17: Time: 24.096448183059692 Total Reward: -35.00000000000071 Avg_Loss: 3.0497909548277615 Episode: 18: Time: 24.02280306816101 Total Reward: -49.45529010238625 Avg_Loss: 1.3030801909803833 Episode: 19: Time: 24.25257110595703 Total Reward: 117.01413427561931 Avg_Loss: 2.6935463223387215 Episode: 20: Time: 23.678778886795044 Total Reward: 71.07773851590402 Avg_Loss: 1.6022877284959585 Episode: 21: Time: 23.742127180099487 Total Reward: -35.19933554817311 Avg_Loss: 2.7878835389093193 Episode: 22: Time: 23.64303994178772 Total Reward: -23.30188679245331 Avg_Loss: 1.7956996214489978 Episode: 23: Time: 24.060755968093872 Total Reward: 305.0000000000021 Avg_Loss: 2.1934938334366856 Episode: 24: Time: 22.312655687332153 Total Reward: -103.19480968858136 Avg_Loss: 2.3625617214043935 Episode: 25: Time: 22.964369773864746 Total Reward: 292.49999999999363 Avg_Loss: 5.594311775780525 Episode: 26: Time: 17.822964906692505 Total Reward: -27.659932659933077 Avg_Loss: 4.938372734464517 Episode: 27: Time: 17.818376779556274 Total Reward: 25.437956204379226 Avg_Loss: 2.1076705150243615 Episode: 28: Time: 17.913553953170776 Total Reward: 186.78694158075987 Avg_Loss: 2.2607203315035638 Episode: 29: Time: 17.98331904411316 Total Reward: 606.8181818181691 Avg_Loss: 2.33925094912533 Episode: 30: Time: 18.062785148620605 Total Reward: 677.5752508361077 Avg_Loss: 2.768643808715484 Episode: 31: Time: 18.17399024963379 Total Reward: 440.836177474389 Avg_Loss: 2.8379861698681568 Episode: 32: Time: 18.691812992095947 Total Reward: 513.5526315789376 Avg_Loss: 4.983561173832717 Episode: 33: Time: 18.72493004798889 Total Reward: 703.5865724381526 Avg_Loss: 5.105834620589969 Episode: 34: Time: 19.065919160842896 Total Reward: 253.75444839856738 Avg_Loss: 6.792224825179877 Episode: 35: Time: 18.82990598678589 Total Reward: 143.85350318471762 Avg_Loss: 4.27089317456013 Episode: 36: Time: 18.75412106513977 Total Reward: 548.3566433566368 Avg_Loss: 5.352592559421764 Episode: 37: Time: 18.695731163024902 Total Reward: 321.3568773234183 Avg_Loss: 4.613672426768711 Episode: 38: Time: 18.875886917114258 Total Reward: 294.5348837209304 Avg_Loss: 6.682771241464534 Episode: 39: Time: 18.508291006088257 Total Reward: 336.3725490195993 Avg_Loss: 6.552195939196258 Episode: 40: Time: 18.377809047698975 Total Reward: -61.66666666666737 Avg_Loss: 6.0563742956694435 Episode: 41: Time: 18.446141242980957 Total Reward: 199.5205479452094 Avg_Loss: 6.441372477707743 Episode: 42: Time: 18.48252010345459 Total Reward: 344.99999999999807 Avg_Loss: 6.266093687350009 Episode: 43: Time: 18.580650806427002 Total Reward: 119.05750798722462 Avg_Loss: 6.28730186995338 Episode: 44: Time: 18.300409078598022 Total Reward: 195.78014184397625 Avg_Loss: 5.177256118850548 Episode: 45: Time: 18.606945991516113 Total Reward: 315.4234527687261 Avg_Loss: 5.987423268185944 Episode: 46: Time: 18.34442901611328 Total Reward: 231.61290322581021 Avg_Loss: 5.979899812145393 Episode: 47: Time: 18.405937910079956 Total Reward: 231.3157894736877 Avg_Loss: 6.190341111992588 Episode: 48: Time: 18.443925619125366 Total Reward: 228.30827067669645 Avg_Loss: 6.239532231783667 Episode: 49: Time: 18.443154096603394 Total Reward: 30.490196078433833 Avg_Loss: 6.175223205269885 Validation Mean Reward: 458.46327438435145 Validation Std Reward: 199.2945111559013 Episode: 50: Time: 18.340068817138672 Total Reward: 145.24024024024064 Avg_Loss: 6.3653465018552895 Episode: 51: Time: 18.21713900566101 Total Reward: 323.06020066889556 Avg_Loss: 6.5958595080535956 Episode: 52: Time: 18.200219869613647 Total Reward: 101.61016949152906 Avg_Loss: 6.2739094105087405 Episode: 53: Time: 18.36238193511963 Total Reward: 113.72274143302627 Avg_Loss: 6.321286172926927 Episode: 54: Time: 18.384412050247192 Total Reward: 175.00000000000244 Avg_Loss: 6.09942497776336 Episode: 55: Time: 18.341345071792603 Total Reward: 360.1724137930996 Avg_Loss: 6.355533613878138 Episode: 56: Time: 18.38343906402588 Total Reward: -40.017182130584885 Avg_Loss: 6.034687754486789 Episode: 57: Time: 18.471060276031494 Total Reward: 223.181818181819 Avg_Loss: 7.0815828963488086 Episode: 58: Time: 18.34193706512451 Total Reward: 283.47222222222183 Avg_Loss: 6.260658145952625 Episode: 59: Time: 18.335744857788086 Total Reward: 237.22591362126437 Avg_Loss: 6.10625439481575 Episode: 60: Time: 18.431900024414062 Total Reward: 150.61403508772284 Avg_Loss: 6.846844851469793 Episode: 61: Time: 18.436804056167603 Total Reward: 329.2424242424203 Avg_Loss: 6.177449177543656 Episode: 62: Time: 18.516599655151367 Total Reward: 119.92537313433013 Avg_Loss: 7.130018908937438 Episode: 63: Time: 18.606340885162354 Total Reward: 308.12499999999335 Avg_Loss: 6.495448444570814 Episode: 64: Time: 18.43500781059265 Total Reward: 207.63157894737057 Avg_Loss: 7.0166590108590965 Episode: 65: Time: 18.426379203796387 Total Reward: 281.23762376237426 Avg_Loss: 6.396317028197922 Episode: 66: Time: 18.391900777816772 Total Reward: 419.17004048582305 Avg_Loss: 6.284059239034893 Episode: 67: Time: 18.309101104736328 Total Reward: 346.0646387832611 Avg_Loss: 6.504040756145446 Episode: 68: Time: 18.300287008285522 Total Reward: 107.53164556962386 Avg_Loss: 6.682994896624269 Episode: 69: Time: 18.486900806427002 Total Reward: 201.1783439490481 Avg_Loss: 6.288084976312493 Episode: 70: Time: 18.304285049438477 Total Reward: 263.6206896551754 Avg_Loss: 6.31439006929638 Episode: 71: Time: 18.757606983184814 Total Reward: 355.3816793893053 Avg_Loss: 5.878656574157106 Episode: 72: Time: 18.706587076187134 Total Reward: 330.0871080139354 Avg_Loss: 6.718670100224118 Episode: 73: Time: 18.5713369846344 Total Reward: 321.66666666666265 Avg_Loss: 6.327078799740607 Episode: 74: Time: 18.614731073379517 Total Reward: 300.2702702702704 Avg_Loss: 6.383947511680987 Episode: 75: Time: 18.68776297569275 Total Reward: 391.3013698630103 Avg_Loss: 6.098800207386498 Episode: 76: Time: 18.367037773132324 Total Reward: 26.794871794873146 Avg_Loss: 5.873700674842386 Episode: 77: Time: 18.382614850997925 Total Reward: 233.30188679245438 Avg_Loss: 6.017036462030491 Episode: 78: Time: 18.69125008583069 Total Reward: 122.94871794872122 Avg_Loss: 6.3378724050121145 Episode: 79: Time: 18.375956058502197 Total Reward: 28.55212355212568 Avg_Loss: 6.427758877016917 Episode: 80: Time: 18.37463903427124 Total Reward: 203.93238434163845 Avg_Loss: 6.334887426941335 Episode: 81: Time: 18.402971982955933 Total Reward: 148.4456928838984 Avg_Loss: 5.9147878494583255 Episode: 82: Time: 18.709431171417236 Total Reward: 311.34920634920485 Avg_Loss: 6.286084405514372
Episode: 83: Time: 18.443274974822998 Total Reward: -37.238267148014856 Avg_Loss: 6.270810764376857 Episode: 84: Time: 18.500017166137695 Total Reward: 254.4809688581352 Avg_Loss: 6.02737505596225 Episode: 85: Time: 18.433401823043823 Total Reward: 298.5018050541516 Avg_Loss: 5.694585790153311 Episode: 86: Time: 18.515270709991455 Total Reward: 339.78260869564576 Avg_Loss: 6.182177367330599 Episode: 87: Time: 19.035494327545166 Total Reward: 401.8152866241946 Avg_Loss: 5.996060750564607 Episode: 88: Time: 18.711929321289062 Total Reward: 370.11627906976213 Avg_Loss: 6.450168994795375 Episode: 89: Time: 19.03132700920105 Total Reward: 265.7594936708896 Avg_Loss: 5.4398660444411915 Episode: 90: Time: 18.665558099746704 Total Reward: 188.27645051194932 Avg_Loss: 5.964842142177229 Episode: 91: Time: 18.57802391052246 Total Reward: 394.5833333333296 Avg_Loss: 6.15013685747355 Episode: 92: Time: 18.88603901863098 Total Reward: 528.6933797909309 Avg_Loss: 6.24037806877569 Episode: 93: Time: 18.658970832824707 Total Reward: 141.93379790941088 Avg_Loss: 6.069298279886486 Episode: 94: Time: 18.5732102394104 Total Reward: 458.43511450381214 Avg_Loss: 5.8717237961392446 Episode: 95: Time: 18.486167669296265 Total Reward: 537.18390804597 Avg_Loss: 5.700710877650926 Episode: 96: Time: 18.526743173599243 Total Reward: 263.1081081081114 Avg_Loss: 6.270226850229151 Episode: 97: Time: 18.83364987373352 Total Reward: 474.9658703071609 Avg_Loss: 6.092408953093681 Episode: 98: Time: 18.67180109024048 Total Reward: 107.53164556962464 Avg_Loss: 5.898113572296976 Episode: 99: Time: 18.695770263671875 Total Reward: 399.16342412450695 Avg_Loss: 6.033409199293922 Validation Mean Reward: 481.8406646195414 Validation Std Reward: 191.37161489843382 Episode: 100: Time: 19.090375900268555 Total Reward: 359.21245421244953 Avg_Loss: 5.846816751135497 Episode: 101: Time: 19.089449167251587 Total Reward: 312.894736842106 Avg_Loss: 6.3626761276181005 Episode: 102: Time: 18.74097776412964 Total Reward: 403.24561403508005 Avg_Loss: 6.156130127546166 Episode: 103: Time: 18.77289891242981 Total Reward: 439.4827586206836 Avg_Loss: 6.089992290785332 Episode: 104: Time: 18.737912893295288 Total Reward: 227.91666666667123 Avg_Loss: 6.064660413926389 Episode: 105: Time: 18.585108995437622 Total Reward: 364.64912280701185 Avg_Loss: 6.128799136947183 Episode: 106: Time: 18.561607122421265 Total Reward: 367.8378378378377 Avg_Loss: 5.972931993107836 Episode: 107: Time: 18.451815843582153 Total Reward: 234.67032967033379 Avg_Loss: 6.012945071989749 Episode: 108: Time: 18.388984203338623 Total Reward: 330.0871080139364 Avg_Loss: 6.5723021120584315 Episode: 109: Time: 18.653986930847168 Total Reward: 208.57142857143225 Avg_Loss: 6.084308267641468 Episode: 110: Time: 18.594002723693848 Total Reward: 326.0526315789448 Avg_Loss: 6.45499836597122 Episode: 111: Time: 18.91716504096985 Total Reward: 313.9456869009548 Avg_Loss: 6.395047280968738 Episode: 112: Time: 18.92250919342041 Total Reward: 476.4285714285635 Avg_Loss: 6.472848514548871 Episode: 113: Time: 18.70432209968567 Total Reward: 287.5503355704722 Avg_Loss: 6.361192389696586 Episode: 114: Time: 18.80393099784851 Total Reward: 212.1895424836647 Avg_Loss: 6.036419403653185 Episode: 115: Time: 18.958651304244995 Total Reward: 429.34456928838193 Avg_Loss: 6.405590161055076 Episode: 116: Time: 19.211135864257812 Total Reward: 464.3220338982992 Avg_Loss: 6.484018000734954 Episode: 117: Time: 19.225416898727417 Total Reward: 341.86006825938375 Avg_Loss: 6.048463645101595 Episode: 118: Time: 19.68211793899536 Total Reward: 440.5805243445627 Avg_Loss: 5.480645969635298 Episode: 119: Time: 19.764899969100952 Total Reward: 160.81395348837677 Avg_Loss: 6.2953638028697805 Episode: 120: Time: 20.286771059036255 Total Reward: 292.09677419354443 Avg_Loss: 6.2510937843002194 Episode: 121: Time: 24.200521230697632 Total Reward: 447.12454212453554 Avg_Loss: 7.0456375725129075 Episode: 122: Time: 25.554591178894043 Total Reward: 601.2962962962863 Avg_Loss: 6.413839215991878 Episode: 123: Time: 25.037989139556885 Total Reward: 358.07443365695406 Avg_Loss: 6.345576865332467 Episode: 124: Time: 24.6795871257782 Total Reward: 461.39097744360026 Avg_Loss: 5.861156826259709 Episode: 125: Time: 24.350960731506348 Total Reward: 533.787878787873 Avg_Loss: 6.113276686988959 Episode: 126: Time: 25.093549251556396 Total Reward: 424.6078431372488 Avg_Loss: 6.315783847279909 Episode: 127: Time: 24.759104013442993 Total Reward: 449.2622950819639 Avg_Loss: 6.858878354064557 Episode: 128: Time: 24.420403003692627 Total Reward: 242.9310344827577 Avg_Loss: 6.657212042007126 Episode: 129: Time: 24.350538969039917 Total Reward: 98.35347432024577 Avg_Loss: 6.283908766858718 Episode: 130: Time: 24.001192808151245 Total Reward: 658.731343283568 Avg_Loss: 6.423150150715804 Episode: 131: Time: 24.24033498764038 Total Reward: 325.3389830508425 Avg_Loss: 5.958090766137388 Episode: 132: Time: 25.2105450630188 Total Reward: 311.1433447098881 Avg_Loss: 6.248263414166555 Episode: 133: Time: 24.595959901809692 Total Reward: 342.4999999999995 Avg_Loss: 6.750374579129099 Episode: 134: Time: 24.489397048950195 Total Reward: 427.6480836236866 Avg_Loss: 7.524032345339029 Episode: 135: Time: 24.619147062301636 Total Reward: 225.58823529412126 Avg_Loss: 6.028418488362256 Episode: 136: Time: 24.267040014266968 Total Reward: 456.99999999999244 Avg_Loss: 6.738868255575164 Episode: 137: Time: 24.290488958358765 Total Reward: 522.4496644295218 Avg_Loss: 6.928866955913415 Episode: 138: Time: 24.619557857513428 Total Reward: 266.2716763005709 Avg_Loss: 6.069157607415143 Episode: 139: Time: 24.261604070663452 Total Reward: 430.47770700636164 Avg_Loss: 6.4664260299265885 Episode: 140: Time: 24.246404886245728 Total Reward: 411.2893081760973 Avg_Loss: 6.635698953596484 Episode: 141: Time: 24.288400888442993 Total Reward: 383.76447876447645 Avg_Loss: 6.68460039030604 Episode: 142: Time: 24.5927836894989 Total Reward: 383.134110787171 Avg_Loss: 6.633279082154026 Episode: 143: Time: 24.187762022018433 Total Reward: 396.408934707899 Avg_Loss: 6.294779628264804 Episode: 144: Time: 24.11202883720398 Total Reward: 501.36363636362455 Avg_Loss: 6.839084355771041 Episode: 145: Time: 24.42151713371277 Total Reward: 528.0769230769179 Avg_Loss: 6.95123922524332 Episode: 146: Time: 24.357394218444824 Total Reward: 497.05776173284016 Avg_Loss: 6.649193443170114 Episode: 147: Time: 24.19088625907898 Total Reward: 325.6642066420551 Avg_Loss: 6.714697713110628 Episode: 148: Time: 24.230807781219482 Total Reward: 404.99999999999835 Avg_Loss: 6.8558993384617715 Episode: 149: Time: 24.220192909240723 Total Reward: 424.6078431372479 Avg_Loss: 6.795134599469289 Validation Mean Reward: 635.5840407064619 Validation Std Reward: 179.53025900865836 Episode: 150: Time: 24.62359619140625 Total Reward: 440.25641025640596 Avg_Loss: 6.618963349266212 Episode: 151: Time: 24.37660026550293 Total Reward: 706.8018018017929 Avg_Loss: 6.343903342214953 Episode: 152: Time: 24.41901206970215 Total Reward: 527.5680933852068 Avg_Loss: 6.08039605767787 Episode: 153: Time: 24.48466992378235 Total Reward: 443.205980066439 Avg_Loss: 6.687250721354444 Episode: 154: Time: 24.42862844467163 Total Reward: 500.87020648966876 Avg_Loss: 6.613719786916461 Episode: 155: Time: 24.442484855651855 Total Reward: 288.21167883210853 Avg_Loss: 6.808082757877703 Episode: 156: Time: 24.924023866653442 Total Reward: 528.2394366197071 Avg_Loss: 6.771831427301679 Episode: 157: Time: 25.144904136657715 Total Reward: 295.09287925696697 Avg_Loss: 7.0549511148148225 Episode: 158: Time: 24.88443088531494 Total Reward: 153.27586206896962 Avg_Loss: 6.781861414428518 Episode: 159: Time: 24.991265058517456 Total Reward: 672.6767676767549 Avg_Loss: 6.961486474806521 Episode: 160: Time: 25.520957946777344 Total Reward: 426.0843373493926 Avg_Loss: 6.968405494169027 Episode: 161: Time: 25.428069829940796 Total Reward: 338.2129963898867 Avg_Loss: 6.9370563681386095 Episode: 162: Time: 25.376903772354126 Total Reward: 440.46099290779307 Avg_Loss: 6.921238874186988 Episode: 163: Time: 25.477674961090088 Total Reward: 342.49999999999966 Avg_Loss: 6.693634323713158 Episode: 164: Time: 24.930181980133057 Total Reward: 458.6332179930704 Avg_Loss: 6.486658299169621
Episode: 165: Time: 24.46898913383484 Total Reward: 717.7340823969928 Avg_Loss: 7.231713039033553 Episode: 166: Time: 24.648165225982666 Total Reward: 306.9933554817293 Avg_Loss: 7.152861952781677 Episode: 167: Time: 25.57377314567566 Total Reward: 599.5337620578658 Avg_Loss: 7.324667753291731 Episode: 168: Time: 25.195861101150513 Total Reward: 404.9999999999983 Avg_Loss: 6.737079249710596 Episode: 169: Time: 25.141915798187256 Total Reward: 175.8333333333383 Avg_Loss: 6.770755751794126 Episode: 170: Time: 24.88740086555481 Total Reward: 544.8601398601315 Avg_Loss: 6.736490658852232 Episode: 171: Time: 24.818866968154907 Total Reward: 253.68421052630913 Avg_Loss: 6.736501869033365 Episode: 172: Time: 25.51734709739685 Total Reward: 496.36212624583777 Avg_Loss: 6.728963853932228 Episode: 173: Time: 25.04335308074951 Total Reward: 430.09652509652193 Avg_Loss: 7.041211037074818 Episode: 174: Time: 24.681636810302734 Total Reward: 503.70550161811246 Avg_Loss: 6.9307246879369275 Episode: 175: Time: 24.626503944396973 Total Reward: 279.21383647798336 Avg_Loss: 7.043648370173799 Episode: 176: Time: 24.62425184249878 Total Reward: 339.40233236151465 Avg_Loss: 6.587687232915093 Episode: 177: Time: 24.36964201927185 Total Reward: 257.7272727272765 Avg_Loss: 6.7523910668717715 Episode: 178: Time: 24.717195987701416 Total Reward: 489.22939068099754 Avg_Loss: 6.604683749815997 Episode: 179: Time: 25.946635723114014 Total Reward: 496.50326797384724 Avg_Loss: 6.260892219904091 Episode: 180: Time: 25.4350368976593 Total Reward: 634.6296296296159 Avg_Loss: 6.606392647037987 Episode: 181: Time: 25.777897119522095 Total Reward: 795.1960784313562 Avg_Loss: 7.160944516418361 Episode: 182: Time: 26.705039978027344 Total Reward: 513.6956521739052 Avg_Loss: 6.871145028527043 Episode: 183: Time: 26.083230018615723 Total Reward: 450.13888888888033 Avg_Loss: 6.621798128640952 Episode: 184: Time: 25.459030628204346 Total Reward: 520.3846153846082 Avg_Loss: 6.45041714796499 Episode: 185: Time: 25.751399040222168 Total Reward: 346.8604651162664 Avg_Loss: 6.713663500898025 Episode: 186: Time: 25.565242052078247 Total Reward: 357.9411764705826 Avg_Loss: 6.877545471952743 Episode: 187: Time: 25.165487051010132 Total Reward: 478.9436619718242 Avg_Loss: 6.501140279930179 Episode: 188: Time: 25.108695030212402 Total Reward: 587.242990654193 Avg_Loss: 7.188233818326678 Episode: 189: Time: 25.019263982772827 Total Reward: 340.4354354354313 Avg_Loss: 6.96229172253809 Episode: 190: Time: 25.470130920410156 Total Reward: 404.99999999999875 Avg_Loss: 6.806157009942191 Episode: 191: Time: 25.191493272781372 Total Reward: 334.50819672131024 Avg_Loss: 6.917553357717369 Episode: 192: Time: 24.560447931289673 Total Reward: 652.3309608540825 Avg_Loss: 7.506783001563129 Episode: 193: Time: 24.27171301841736 Total Reward: 252.98534798533686 Avg_Loss: 6.990330581905461 Episode: 194: Time: 24.46337914466858 Total Reward: 541.6906474820033 Avg_Loss: 7.4378242903396865 Episode: 195: Time: 24.301411867141724 Total Reward: 191.79245283019262 Avg_Loss: 6.740302816158583 Episode: 196: Time: 24.770639896392822 Total Reward: 470.9722222222129 Avg_Loss: 6.733842397938256 Episode: 197: Time: 24.60128688812256 Total Reward: 702.9094076654928 Avg_Loss: 7.378776660987309 Episode: 198: Time: 24.28277087211609 Total Reward: 370.3846153846095 Avg_Loss: 7.644859796812554 Episode: 199: Time: 24.16348910331726 Total Reward: 412.2463768115899 Avg_Loss: 6.8960049733394335 Validation Mean Reward: 519.3207880555005 Validation Std Reward: 164.40497608323466 Test Mean Reward: 587.7975934192326 Test Std Reward: 139.95309744754275
([0.7032929947659089, 0.6477146975758696, 0.5800422116832322, 0.6849102419370613, 0.7501259394177869, 0.7486689495658424, 0.8814797621815145, 0.9176623095609561, 0.871715443678388, 0.9815360074784575, 1.0410380279567062, 1.1218535047368843, 1.2063556467848164, 1.27918864619233, 4.027322133627316, 4.554474374764607, 3.0497909548277615, 1.3030801909803833, 2.6935463223387215, 1.6022877284959585, 2.7878835389093193, 1.7956996214489978, 2.1934938334366856, 2.3625617214043935, 5.594311775780525, 4.938372734464517, 2.1076705150243615, 2.2607203315035638, 2.33925094912533, 2.768643808715484, 2.8379861698681568, 4.983561173832717, 5.105834620589969, 6.792224825179877, 4.27089317456013, 5.352592559421764, 4.613672426768711, 6.682771241464534, 6.552195939196258, 6.0563742956694435, 6.441372477707743, 6.266093687350009, 6.28730186995338, 5.177256118850548, 5.987423268185944, 5.979899812145393, 6.190341111992588, 6.239532231783667, 6.175223205269885, 6.3653465018552895, 6.5958595080535956, 6.2739094105087405, 6.321286172926927, 6.09942497776336, 6.355533613878138, 6.034687754486789, 7.0815828963488086, 6.260658145952625, 6.10625439481575, 6.846844851469793, 6.177449177543656, 7.130018908937438, 6.495448444570814, 7.0166590108590965, 6.396317028197922, 6.284059239034893, 6.504040756145446, 6.682994896624269, 6.288084976312493, 6.31439006929638, 5.878656574157106, 6.718670100224118, 6.327078799740607, 6.383947511680987, 6.098800207386498, 5.873700674842386, 6.017036462030491, 6.3378724050121145, 6.427758877016917, 6.334887426941335, 5.9147878494583255, 6.286084405514372, 6.270810764376857, 6.02737505596225, 5.694585790153311, 6.182177367330599, 5.996060750564607, 6.450168994795375, 5.4398660444411915, 5.964842142177229, 6.15013685747355, 6.24037806877569, 6.069298279886486, 5.8717237961392446, 5.700710877650926, 6.270226850229151, 6.092408953093681, 5.898113572296976, 6.033409199293922, 5.846816751135497, 6.3626761276181005, 6.156130127546166, 6.089992290785332, 6.064660413926389, 6.128799136947183, 5.972931993107836, 6.012945071989749, 6.5723021120584315, 6.084308267641468, 6.45499836597122, 6.395047280968738, 6.472848514548871, 6.361192389696586, 6.036419403653185, 6.405590161055076, 6.484018000734954, 6.048463645101595, 5.480645969635298, 6.2953638028697805, 6.2510937843002194, 7.0456375725129075, 6.413839215991878, 6.345576865332467, 5.861156826259709, 6.113276686988959, 6.315783847279909, 6.858878354064557, 6.657212042007126, 6.283908766858718, 6.423150150715804, 5.958090766137388, 6.248263414166555, 6.750374579129099, 7.524032345339029, 6.028418488362256, 6.738868255575164, 6.928866955913415, 6.069157607415143, 6.4664260299265885, 6.635698953596484, 6.68460039030604, 6.633279082154026, 6.294779628264804, 6.839084355771041, 6.95123922524332, 6.649193443170114, 6.714697713110628, 6.8558993384617715, 6.795134599469289, 6.618963349266212, 6.343903342214953, 6.08039605767787, 6.687250721354444, 6.613719786916461, 6.808082757877703, 6.771831427301679, 7.0549511148148225, 6.781861414428518, 6.961486474806521, 6.968405494169027, 6.9370563681386095, 6.921238874186988, 6.693634323713158, 6.486658299169621, 7.231713039033553, 7.152861952781677, 7.324667753291731, 6.737079249710596, 6.770755751794126, 6.736490658852232, 6.736501869033365, 6.728963853932228, 7.041211037074818, 6.9307246879369275, 7.043648370173799, 6.587687232915093, 6.7523910668717715, 6.604683749815997, 6.260892219904091, 6.606392647037987, 7.160944516418361, 6.871145028527043, 6.621798128640952, 6.45041714796499, 6.713663500898025, 6.877545471952743, 6.501140279930179, 7.188233818326678, 6.96229172253809, 6.806157009942191, 6.917553357717369, 7.506783001563129, 6.990330581905461, 7.4378242903396865, 6.740302816158583, 6.733842397938256, 7.378776660987309, 7.644859796812554, 6.8960049733394335], [-44.64028776978459, -56.0389610389619, -59.78873239436666, -5.394265232975001, -32.72893772893818, -6.184210526316598, 36.94444444444324, -33.628158844765835, -12.293233082707298, -26.8996415770615, -36.55844155844217, -18.87543252595222, -34.28571428571476, -150.19595959596012, -139.13174603174642, -42.530864197531216, -35.00000000000071, -49.45529010238625, 117.01413427561931, 71.07773851590402, -35.19933554817311, -23.30188679245331, 305.0000000000021, -103.19480968858136, 292.49999999999363, -27.659932659933077, 25.437956204379226, 186.78694158075987, 606.8181818181691, 677.5752508361077, 440.836177474389, 513.5526315789376, 703.5865724381526, 253.75444839856738, 143.85350318471762, 548.3566433566368, 321.3568773234183, 294.5348837209304, 336.3725490195993, -61.66666666666737, 199.5205479452094, 344.99999999999807, 119.05750798722462, 195.78014184397625, 315.4234527687261, 231.61290322581021, 231.3157894736877, 228.30827067669645, 30.490196078433833, 145.24024024024064, 323.06020066889556, 101.61016949152906, 113.72274143302627, 175.00000000000244, 360.1724137930996, -40.017182130584885, 223.181818181819, 283.47222222222183, 237.22591362126437, 150.61403508772284, 329.2424242424203, 119.92537313433013, 308.12499999999335, 207.63157894737057, 281.23762376237426, 419.17004048582305, 346.0646387832611, 107.53164556962386, 201.1783439490481, 263.6206896551754, 355.3816793893053, 330.0871080139354, 321.66666666666265, 300.2702702702704, 391.3013698630103, 26.794871794873146, 233.30188679245438, 122.94871794872122, 28.55212355212568, 203.93238434163845, 148.4456928838984, 311.34920634920485, -37.238267148014856, 254.4809688581352, 298.5018050541516, 339.78260869564576, 401.8152866241946, 370.11627906976213, 265.7594936708896, 188.27645051194932, 394.5833333333296, 528.6933797909309, 141.93379790941088, 458.43511450381214, 537.18390804597, 263.1081081081114, 474.9658703071609, 107.53164556962464, 399.16342412450695, 359.21245421244953, 312.894736842106, 403.24561403508005, 439.4827586206836, 227.91666666667123, 364.64912280701185, 367.8378378378377, 234.67032967033379, 330.0871080139364, 208.57142857143225, 326.0526315789448, 313.9456869009548, 476.4285714285635, 287.5503355704722, 212.1895424836647, 429.34456928838193, 464.3220338982992, 341.86006825938375, 440.5805243445627, 160.81395348837677, 292.09677419354443, 447.12454212453554, 601.2962962962863, 358.07443365695406, 461.39097744360026, 533.787878787873, 424.6078431372488, 449.2622950819639, 242.9310344827577, 98.35347432024577, 658.731343283568, 325.3389830508425, 311.1433447098881, 342.4999999999995, 427.6480836236866, 225.58823529412126, 456.99999999999244, 522.4496644295218, 266.2716763005709, 430.47770700636164, 411.2893081760973, 383.76447876447645, 383.134110787171, 396.408934707899, 501.36363636362455, 528.0769230769179, 497.05776173284016, 325.6642066420551, 404.99999999999835, 424.6078431372479, 440.25641025640596, 706.8018018017929, 527.5680933852068, 443.205980066439, 500.87020648966876, 288.21167883210853, 528.2394366197071, 295.09287925696697, 153.27586206896962, 672.6767676767549, 426.0843373493926, 338.2129963898867, 440.46099290779307, 342.49999999999966, 458.6332179930704, 717.7340823969928, 306.9933554817293, 599.5337620578658, 404.9999999999983, 175.8333333333383, 544.8601398601315, 253.68421052630913, 496.36212624583777, 430.09652509652193, 503.70550161811246, 279.21383647798336, 339.40233236151465, 257.7272727272765, 489.22939068099754, 496.50326797384724, 634.6296296296159, 795.1960784313562, 513.6956521739052, 450.13888888888033, 520.3846153846082, 346.8604651162664, 357.9411764705826, 478.9436619718242, 587.242990654193, 340.4354354354313, 404.99999999999875, 334.50819672131024, 652.3309608540825, 252.98534798533686, 541.6906474820033, 191.79245283019262, 470.9722222222129, 702.9094076654928, 370.3846153846095, 412.2463768115899], [458.46327438435145, 481.8406646195414, 635.5840407064619, 519.3207880555005], [199.2945111559013, 191.37161489843382, 179.53025900865836, 164.40497608323466])
xxxxxxxxxxtotal_rewards, frames = traineSoftUpdateDQN.play_episode(0,True,42)anim = animate(frames)HTML(anim.to_jshtml())xxxxxxxxxx#### Questions:- Which method performed better? (5 points)- If we modify the $\tau$ for soft updates or the $C$ for the hard updates, how does this affect the performance of the model, come up with a intuition for this, then experimentally verify this. (5 points)xxxxxxxxxx**Ans.** - DQN with soft updates > vanila DQN > DQN with hard updates.- Raising 𝐶 reduces updates, speeds up training but risks outdated networks, slower convergence, and less accurate Q-value estimates. Increasing 𝜏 in Soft Updates slows updates, therefore, enhances stability but risks slower convergence and adaptation. As you can see below, we increased C for hard updates, leading to a slower convergence but better results than before. We also increased 𝜏 in soft updates, resulting in an increase with the stability of the model, slower updates, and also better results.Ans.
xxxxxxxxxxtrainerHardUpdateDQN = DQN.HardUpdateDQN(EnvWrapper(env), model.Nature_Paper_Conv, update_freq = 200, lr = 0.00025, gamma = 0.95, buffer_size=100000, batch_size=32, loss_fn = "mse_loss", use_wandb = False, device = 'cpu', seed = 42, epsilon_scheduler = utils.exponential_decay(1, 1000,0.1), save_path = utils.get_save_path("DoubleDQN_HardUpdates/","./runs/"))trainerHardUpdateDQN.train(200,50,30,50,50)saving to ./runs/DoubleDQN_HardUpdates/run3 Episode: 1: Time: 18.100610971450806 Total Reward: -31.708860759494037 Avg_Loss: 0.820937740907168 Episode: 2: Time: 24.25102162361145 Total Reward: -40.762711864407095 Avg_Loss: 0.8298551891969532 Episode: 3: Time: 24.652439832687378 Total Reward: -41.20253164557029 Avg_Loss: 0.7261427032346485 Episode: 4: Time: 24.62842631340027 Total Reward: -43.08743169398939 Avg_Loss: 0.678266048055737 Episode: 5: Time: 24.391802072525024 Total Reward: -37.942942942943475 Avg_Loss: 0.6597021974451026 Episode: 6: Time: 24.475948810577393 Total Reward: -32.95620437956265 Avg_Loss: 0.6943003801988954 Episode: 7: Time: 24.12496304512024 Total Reward: -30.06493506493574 Avg_Loss: 0.7574385171051786 Episode: 8: Time: 24.135298252105713 Total Reward: -30.251798561151407 Avg_Loss: 0.6477190712858149 Episode: 9: Time: 24.13827419281006 Total Reward: -85.25974025973977 Avg_Loss: 0.641771079080195 Episode: 10: Time: 27.539865255355835 Total Reward: -63.30985915493014 Avg_Loss: 0.6105592137656543 Episode: 11: Time: 27.32367515563965 Total Reward: -16.14695340501838 Avg_Loss: 0.6308907887765339 Episode: 12: Time: 26.78989887237549 Total Reward: -25.402930402931116 Avg_Loss: 0.6612010868031437 Episode: 13: Time: 27.596198081970215 Total Reward: 33.289473684209526 Avg_Loss: 0.7037577274247628 Episode: 14: Time: 27.20631194114685 Total Reward: -22.083333333334075 Avg_Loss: 0.7539445211437821 Episode: 15: Time: 26.99581003189087 Total Reward: -30.01805054151692 Avg_Loss: 0.784094048930066 Episode: 16: Time: 27.199682235717773 Total Reward: -31.090225563910515 Avg_Loss: 0.7451601366849006 Episode: 17: Time: 27.021485090255737 Total Reward: -26.899641577061658 Avg_Loss: 0.7497997194716409 Episode: 18: Time: 27.556822061538696 Total Reward: -13.831168831169647 Avg_Loss: 0.862367572028096 Episode: 19: Time: 27.500248908996582 Total Reward: 8.806228373702739 Avg_Loss: 0.9059205599943129 Episode: 20: Time: 27.523337364196777 Total Reward: 51.42857142857394 Avg_Loss: 1.0743605771861156 Episode: 21: Time: 27.56849694252014 Total Reward: 228.23232323232378 Avg_Loss: 1.2333044959466999 Episode: 22: Time: 27.871034860610962 Total Reward: -37.85714285714354 Avg_Loss: 1.3613839353896489 Episode: 23: Time: 24.732903957366943 Total Reward: -5.493827160494616 Avg_Loss: 1.299452933130645 Episode: 24: Time: 24.125608205795288 Total Reward: -5.000000000000909 Avg_Loss: 1.1972777161966353 Episode: 25: Time: 24.081329822540283 Total Reward: 273.60068259385565 Avg_Loss: 1.3706221135348833 Episode: 26: Time: 24.09641194343567 Total Reward: 141.74911660777806 Avg_Loss: 1.5413459511865086 Episode: 27: Time: 24.076580286026 Total Reward: 78.14487632509119 Avg_Loss: 1.854454691670522 Episode: 28: Time: 24.114107847213745 Total Reward: 24.60132890365381 Avg_Loss: 1.9144588631241262 Episode: 29: Time: 24.213094234466553 Total Reward: 188.01886792453118 Avg_Loss: 2.0782367310353687 Episode: 30: Time: 24.53823494911194 Total Reward: 9.615384615383997 Avg_Loss: 2.033498555672269 Episode: 31: Time: 24.34040403366089 Total Reward: 420.5709342560441 Avg_Loss: 2.2022186645439694 Episode: 32: Time: 26.278673887252808 Total Reward: 180.00000000000392 Avg_Loss: 2.584135141442804 Episode: 33: Time: 27.681531190872192 Total Reward: 268.6363636363591 Avg_Loss: 2.7975101751439713 Episode: 34: Time: 27.5102801322937 Total Reward: 182.372262773726 Avg_Loss: 3.13859794044695 Episode: 35: Time: 27.96111798286438 Total Reward: 197.09621993127547 Avg_Loss: 2.9012930012550675 Episode: 36: Time: 27.89676284790039 Total Reward: 388.6363636363559 Avg_Loss: 2.7518174052238464 Episode: 37: Time: 27.69805908203125 Total Reward: 102.32441471572366 Avg_Loss: 2.602452131629992 Episode: 38: Time: 27.281780004501343 Total Reward: 7.3890784982942606 Avg_Loss: 2.490580059149686 Episode: 39: Time: 27.58149480819702 Total Reward: 279.9999999999965 Avg_Loss: 2.676145635232204 Episode: 40: Time: 27.39803695678711 Total Reward: 198.28621908127596 Avg_Loss: 3.0447231716468552 Episode: 41: Time: 27.607063055038452 Total Reward: 196.81494661922162 Avg_Loss: 3.0021196769065215 Episode: 42: Time: 27.550299882888794 Total Reward: 252.1337579617767 Avg_Loss: 3.218664873422695 Episode: 43: Time: 27.53882598876953 Total Reward: 338.5664335664332 Avg_Loss: 3.4549734439669537 Episode: 44: Time: 27.70739507675171 Total Reward: 410.5762081784316 Avg_Loss: 3.540909213178298 Episode: 45: Time: 27.864111185073853 Total Reward: 3.8372093023251868 Avg_Loss: 3.669427632784643 Episode: 46: Time: 27.621251821517944 Total Reward: 94.54248366013526 Avg_Loss: 3.413881740650209 Episode: 47: Time: 27.787701845169067 Total Reward: 225.0000000000045 Avg_Loss: 3.255634562308047 Episode: 48: Time: 27.604851007461548 Total Reward: 343.35616438355765 Avg_Loss: 3.78113797682674 Episode: 49: Time: 27.416562795639038 Total Reward: 148.636363636366 Avg_Loss: 3.7284614934640774 Validation Mean Reward: 201.8575741131731 Validation Std Reward: 160.6668853162569 Episode: 50: Time: 27.152767181396484 Total Reward: 340.9999999999974 Avg_Loss: 4.425017001749087 Episode: 51: Time: 27.513978958129883 Total Reward: 358.90070921985466 Avg_Loss: 4.301955237608998 Episode: 52: Time: 27.59372115135193 Total Reward: 184.2792792792833 Avg_Loss: 4.302668186546374 Episode: 53: Time: 29.03341293334961 Total Reward: 127.58064516129335 Avg_Loss: 4.197580825380919 Episode: 54: Time: 24.716553926467896 Total Reward: 418.6186770427905 Avg_Loss: 4.513498431494256 Episode: 55: Time: 24.523921728134155 Total Reward: 389.3205574912811 Avg_Loss: 4.808122724044223 Episode: 56: Time: 24.552629947662354 Total Reward: 160.5555555555564 Avg_Loss: 4.391635022243531 Episode: 57: Time: 24.469712734222412 Total Reward: 235.33033033033084 Avg_Loss: 4.361524848627443 Episode: 58: Time: 24.629624128341675 Total Reward: 242.79264214047328 Avg_Loss: 4.647138198383716 Episode: 59: Time: 24.569458961486816 Total Reward: 277.88135593219675 Avg_Loss: 4.402270266488821 Episode: 60: Time: 24.797054052352905 Total Reward: 188.48909657321346 Avg_Loss: 4.364726959657268 Episode: 61: Time: 24.608376026153564 Total Reward: 275.0000000000034 Avg_Loss: 4.116379081201153 Episode: 62: Time: 24.491047859191895 Total Reward: 432.5862068965424 Avg_Loss: 4.242655820205432 Episode: 63: Time: 24.444114923477173 Total Reward: 317.37113402061823 Avg_Loss: 4.650930493819613 Episode: 64: Time: 24.43746781349182 Total Reward: 212.69230769231112 Avg_Loss: 4.646883855847752 Episode: 65: Time: 24.20148992538452 Total Reward: 5.694444444443539 Avg_Loss: 4.924998957569859 Episode: 66: Time: 24.284271240234375 Total Reward: 290.3820598006585 Avg_Loss: 4.544403590574986 Episode: 67: Time: 24.243502140045166 Total Reward: 304.9999999999919 Avg_Loss: 4.566294788813391 Episode: 68: Time: 24.346744060516357 Total Reward: 322.5084175084156 Avg_Loss: 4.180092493526074 Episode: 69: Time: 24.58715796470642 Total Reward: 218.43283582089936 Avg_Loss: 4.257498892916351 Episode: 70: Time: 24.420358180999756 Total Reward: 226.87500000000426 Avg_Loss: 4.409875603283153 Episode: 71: Time: 24.29133892059326 Total Reward: 368.8157894736778 Avg_Loss: 4.501948399203164 Episode: 72: Time: 24.244570016860962 Total Reward: 297.739273927393 Avg_Loss: 4.583714746126608 Episode: 73: Time: 24.139886140823364 Total Reward: 204.5951417004059 Avg_Loss: 4.999942636289516 Episode: 74: Time: 24.10243511199951 Total Reward: 380.2851711026554 Avg_Loss: 4.677865306369397 Episode: 75: Time: 24.385620832443237 Total Reward: 237.2784810126617 Avg_Loss: 4.841272954680338 Episode: 76: Time: 24.262428998947144 Total Reward: 137.4840764331252 Avg_Loss: 4.446507530553 Episode: 77: Time: 24.168675184249878 Total Reward: 173.96551724138106 Avg_Loss: 4.844482975346701 Episode: 78: Time: 24.230176210403442 Total Reward: 309.5801526717537 Avg_Loss: 4.84062044860936 Episode: 79: Time: 24.126905918121338 Total Reward: 371.8989547038308 Avg_Loss: 5.348679126310749 Episode: 80: Time: 24.234661102294922 Total Reward: 148.33333333333738 Avg_Loss: 4.451933546226566 Episode: 81: Time: 24.27575397491455 Total Reward: 252.9729729729689 Avg_Loss: 4.64440794351722 Episode: 82: Time: 24.156310081481934 Total Reward: 315.95890410958356 Avg_Loss: 5.2193897166172
Episode: 83: Time: 24.205204010009766 Total Reward: 286.41025641025595 Avg_Loss: 4.961310871008064 Episode: 84: Time: 6.477529048919678 Total Reward: -61.44905660377387 Avg_Loss: 5.2851335305434 Episode: 85: Time: 24.353999137878418 Total Reward: 151.79487179487413 Avg_Loss: 8.26195216930213 Episode: 86: Time: 24.08628010749817 Total Reward: 28.55212355212548 Avg_Loss: 5.074071414330426 Episode: 87: Time: 24.042032957077026 Total Reward: 342.7224199288254 Avg_Loss: 4.938717095791793 Episode: 88: Time: 24.03946805000305 Total Reward: 425.5992509363207 Avg_Loss: 5.031276339743318 Episode: 89: Time: 24.244354009628296 Total Reward: 241.50793650793923 Avg_Loss: 4.914846381720374 Episode: 90: Time: 24.0889310836792 Total Reward: 500.6678700360931 Avg_Loss: 5.255969357089836 Episode: 91: Time: 24.211157083511353 Total Reward: 337.5259515570859 Avg_Loss: 6.335310681026523 Episode: 92: Time: 24.040297985076904 Total Reward: 392.3646209386246 Avg_Loss: 5.407486358610522 Episode: 93: Time: 24.102522134780884 Total Reward: 212.97101449275738 Avg_Loss: 5.353176939387281 Episode: 94: Time: 24.219416856765747 Total Reward: 220.2866242038258 Avg_Loss: 6.522208352048858 Episode: 95: Time: 24.20154333114624 Total Reward: 413.30564784052433 Avg_Loss: 6.568047005589269 Episode: 96: Time: 24.250727891921997 Total Reward: 382.8481012658181 Avg_Loss: 5.1776827678960915 Episode: 97: Time: 24.204493045806885 Total Reward: 317.96928327645077 Avg_Loss: 5.279811822065787 Episode: 98: Time: 24.221740007400513 Total Reward: 450.1388888888868 Avg_Loss: 5.313009090283337 Episode: 99: Time: 24.11687707901001 Total Reward: 382.35191637630163 Avg_Loss: 4.862757809021893 Validation Mean Reward: 445.9416159286664 Validation Std Reward: 163.82271763193447 Episode: 100: Time: 24.57162117958069 Total Reward: 367.897526501758 Avg_Loss: 4.713422799310765 Episode: 101: Time: 24.36724019050598 Total Reward: 326.64179104477273 Avg_Loss: 4.990601285665977 Episode: 102: Time: 24.348431825637817 Total Reward: 342.69968051117945 Avg_Loss: 4.774985970569258 Episode: 103: Time: 24.197349071502686 Total Reward: 453.5074626865611 Avg_Loss: 5.171170523687571 Episode: 104: Time: 24.17732524871826 Total Reward: 397.7007299270019 Avg_Loss: 5.379169203654057 Episode: 105: Time: 24.004597902297974 Total Reward: 93.40579710145292 Avg_Loss: 5.629865767575112 Episode: 106: Time: 24.119940042495728 Total Reward: 456.85185185184423 Avg_Loss: 5.378657738200757 Episode: 107: Time: 24.116477012634277 Total Reward: 322.5824175824099 Avg_Loss: 5.383543312048712 Episode: 108: Time: 24.270435094833374 Total Reward: 66.18421052631804 Avg_Loss: 5.486103039829671 Episode: 109: Time: 24.2054340839386 Total Reward: 368.1578947368381 Avg_Loss: 5.794330969578078 Episode: 110: Time: 24.26224398612976 Total Reward: 353.27586206896274 Avg_Loss: 5.099011708207491 Episode: 111: Time: 24.32675814628601 Total Reward: 300.83333333332536 Avg_Loss: 5.722746386998842 Episode: 112: Time: 24.303010940551758 Total Reward: 375.17543859648663 Avg_Loss: 6.2354671404141335 Episode: 113: Time: 24.43296217918396 Total Reward: 327.2972972972949 Avg_Loss: 5.5336092680442235 Episode: 114: Time: 24.289695978164673 Total Reward: 381.190476190472 Avg_Loss: 5.137194256822602 Episode: 115: Time: 24.297413110733032 Total Reward: 455.5226480836172 Avg_Loss: 4.9802546240702394 Episode: 116: Time: 24.521008253097534 Total Reward: 146.071428571432 Avg_Loss: 5.230214607815783 Episode: 117: Time: 24.42304515838623 Total Reward: 189.21052631579312 Avg_Loss: 5.346893361135691 Episode: 118: Time: 24.653837203979492 Total Reward: 151.00638977636208 Avg_Loss: 5.562489731972959 Episode: 119: Time: 24.293694972991943 Total Reward: 319.2857142857122 Avg_Loss: 5.90870685737674 Episode: 120: Time: 24.340561151504517 Total Reward: 522.4496644295259 Avg_Loss: 5.467991328539968 Episode: 121: Time: 24.841398000717163 Total Reward: 326.56862745097715 Avg_Loss: 5.37709853829456 Episode: 122: Time: 24.686907052993774 Total Reward: 227.0973782771575 Avg_Loss: 5.1312863911901205 Episode: 123: Time: 19.23439621925354 Total Reward: 349.06779661016003 Avg_Loss: 5.593012961018987 Episode: 124: Time: 19.006242036819458 Total Reward: 85.88737201365646 Avg_Loss: 5.43684516584172 Episode: 125: Time: 19.01735520362854 Total Reward: 302.00374531834535 Avg_Loss: 5.4918706767699295 Episode: 126: Time: 19.902890920639038 Total Reward: 443.7596899224762 Avg_Loss: 5.841477891978095 Episode: 127: Time: 20.2727370262146 Total Reward: 411.451612903221 Avg_Loss: 5.773884406109818 Episode: 128: Time: 19.714649200439453 Total Reward: 436.135531135522 Avg_Loss: 5.462210106248615 Episode: 129: Time: 19.587775945663452 Total Reward: 456.85185185183855 Avg_Loss: 5.614704013371668 Episode: 130: Time: 19.628441095352173 Total Reward: 432.5080906148812 Avg_Loss: 5.689391327505352 Episode: 131: Time: 19.374846935272217 Total Reward: 427.5563909774394 Avg_Loss: 5.709828709854799 Episode: 132: Time: 19.323538064956665 Total Reward: 124.69696969697227 Avg_Loss: 6.069051140496711 Episode: 133: Time: 19.185903072357178 Total Reward: 287.3529411764724 Avg_Loss: 5.868505199416345 Episode: 134: Time: 19.175248861312866 Total Reward: 275.4918032786851 Avg_Loss: 5.795209387270343 Episode: 135: Time: 19.255171060562134 Total Reward: 287.75862068964636 Avg_Loss: 5.797882557917042 Episode: 136: Time: 19.185537099838257 Total Reward: 306.81268882175084 Avg_Loss: 5.962688826713242 Episode: 137: Time: 18.887367725372314 Total Reward: 733.3582089552098 Avg_Loss: 5.61739564342659 Episode: 138: Time: 19.099467992782593 Total Reward: 450.7627118644 Avg_Loss: 5.707876933722937 Episode: 139: Time: 18.94212770462036 Total Reward: 307.73037542661507 Avg_Loss: 6.475288034487171 Episode: 140: Time: 18.823678970336914 Total Reward: 105.65789473684507 Avg_Loss: 5.259320781511419 Episode: 141: Time: 19.05629801750183 Total Reward: 493.85017421601646 Avg_Loss: 5.713927576521866 Episode: 142: Time: 19.080024003982544 Total Reward: 213.82352941176848 Avg_Loss: 5.316184337399587 Episode: 143: Time: 18.688671827316284 Total Reward: 476.99999999999216 Avg_Loss: 5.938543642268462 Episode: 144: Time: 18.654282093048096 Total Reward: 237.21476510067126 Avg_Loss: 5.671118972682152 Episode: 145: Time: 19.009522199630737 Total Reward: 387.6589595375691 Avg_Loss: 5.597070894321473 Episode: 146: Time: 18.79818892478943 Total Reward: 188.43949044586367 Avg_Loss: 5.46676346133737 Episode: 147: Time: 18.90554690361023 Total Reward: 455.3144654087988 Avg_Loss: 5.292451065628469 Episode: 148: Time: 18.675209283828735 Total Reward: 488.0115830115729 Avg_Loss: 5.058371685132259 Episode: 149: Time: 18.98654794692993 Total Reward: 181.96793002915885 Avg_Loss: 5.165304880683162 Validation Mean Reward: 376.4456516851086 Validation Std Reward: 140.30196664980343 Episode: 150: Time: 19.121129989624023 Total Reward: 428.02631578946574 Avg_Loss: 5.594748966834125 Episode: 151: Time: 18.912548065185547 Total Reward: 452.2312703583 Avg_Loss: 5.644749094458187 Episode: 152: Time: 18.8489990234375 Total Reward: 426.3114754098273 Avg_Loss: 5.8408784756139545 Episode: 153: Time: 18.702422857284546 Total Reward: 482.9467680608253 Avg_Loss: 5.450479232964396 Episode: 154: Time: 18.768694162368774 Total Reward: 304.22480620154926 Avg_Loss: 4.905775434830609 Episode: 155: Time: 18.855607748031616 Total Reward: 240.74007220217027 Avg_Loss: 5.367668441363743 Episode: 156: Time: 18.71768879890442 Total Reward: 488.3333333333272 Avg_Loss: 5.321975791153788 Episode: 157: Time: 18.846483945846558 Total Reward: 401.79487179486443 Avg_Loss: 5.308318919995251 Episode: 158: Time: 18.703874111175537 Total Reward: 134.7297297297336 Avg_Loss: 5.4092133826568345 Episode: 159: Time: 18.738615036010742 Total Reward: 196.82879377432312 Avg_Loss: 5.396987089088985 Episode: 160: Time: 19.052348136901855 Total Reward: 546.1960132890242 Avg_Loss: 5.0815691832734755 Episode: 161: Time: 19.146767139434814 Total Reward: 497.92035398229496 Avg_Loss: 5.425141758277636 Episode: 162: Time: 19.00976800918579 Total Reward: 328.3576642335696 Avg_Loss: 5.237884370218806 Episode: 163: Time: 19.066184043884277 Total Reward: 436.6901408450667 Avg_Loss: 5.445181433894053 Episode: 164: Time: 19.18257784843445 Total Reward: 335.3405572755354 Avg_Loss: 5.611861747353017
Episode: 165: Time: 19.107198238372803 Total Reward: 394.6551724137886 Avg_Loss: 5.902027753220887 Episode: 166: Time: 19.161369800567627 Total Reward: 453.821548821536 Avg_Loss: 5.636468718532755 Episode: 167: Time: 19.289504051208496 Total Reward: 377.89156626505496 Avg_Loss: 6.263388074245773 Episode: 168: Time: 19.11868405342102 Total Reward: 616.1913357400624 Avg_Loss: 6.873990818732927 Episode: 169: Time: 19.13919997215271 Total Reward: 419.1843971631158 Avg_Loss: 6.404269650202839 Episode: 170: Time: 19.18834090232849 Total Reward: 460.92105263156884 Avg_Loss: 6.263270647585893 Episode: 171: Time: 19.113001108169556 Total Reward: 261.401384083048 Avg_Loss: 6.352828298296247 Episode: 172: Time: 18.982282876968384 Total Reward: 354.43820224718337 Avg_Loss: 6.045260823073507 Episode: 173: Time: 18.913387060165405 Total Reward: 240.54817275747786 Avg_Loss: 6.507352567520462 Episode: 174: Time: 19.300755262374878 Total Reward: 342.29903536977037 Avg_Loss: 6.223971242664241 Episode: 175: Time: 19.05421805381775 Total Reward: 468.8297872340382 Avg_Loss: 6.358647546848329 Episode: 176: Time: 19.034499168395996 Total Reward: 345.97222222221814 Avg_Loss: 6.103919041256945 Episode: 177: Time: 19.026664972305298 Total Reward: 478.4265734265687 Avg_Loss: 6.29172532999215 Episode: 178: Time: 18.93662977218628 Total Reward: 401.71052631578084 Avg_Loss: 6.297825663029647 Episode: 179: Time: 19.358162879943848 Total Reward: 532.9069767441736 Avg_Loss: 6.3850349132754225 Episode: 180: Time: 19.145837783813477 Total Reward: 464.84555984555345 Avg_Loss: 6.297533519127789 Episode: 181: Time: 19.17695379257202 Total Reward: 455.1618122977303 Avg_Loss: 5.746362517861759 Episode: 182: Time: 19.160319805145264 Total Reward: 244.6226415094381 Avg_Loss: 5.627700875286295 Episode: 183: Time: 19.353153228759766 Total Reward: 281.0932944606431 Avg_Loss: 5.743352380119452 Episode: 184: Time: 19.096107006072998 Total Reward: 345.0000000000007 Avg_Loss: 5.688690711470211 Episode: 185: Time: 19.161567211151123 Total Reward: 449.802867383509 Avg_Loss: 5.232670697845331 Episode: 186: Time: 19.11700415611267 Total Reward: 434.41176470587476 Avg_Loss: 5.262674479424453 Episode: 187: Time: 19.12030291557312 Total Reward: 519.8148148148039 Avg_Loss: 5.1694381036678285 Episode: 188: Time: 18.97779107093811 Total Reward: 42.25490196078759 Avg_Loss: 5.222657639439366 Episode: 189: Time: 19.361706972122192 Total Reward: 423.39464882942525 Avg_Loss: 5.378060609853568 Episode: 190: Time: 19.11591601371765 Total Reward: 516.1111111111027 Avg_Loss: 5.652116600204916 Episode: 191: Time: 19.279432773590088 Total Reward: 469.102564102555 Avg_Loss: 5.619969391021408 Episode: 192: Time: 19.4111430644989 Total Reward: 390.0498338870405 Avg_Loss: 6.401196401660182 Episode: 193: Time: 20.765865325927734 Total Reward: 334.41176470587595 Avg_Loss: 5.665336586848027 Episode: 194: Time: 19.43436908721924 Total Reward: 376.83098591548776 Avg_Loss: 5.662818333681892 Episode: 195: Time: 19.254071712493896 Total Reward: 297.5233644859756 Avg_Loss: 5.443334556427322 Episode: 196: Time: 19.512159824371338 Total Reward: 268.36336336336586 Avg_Loss: 5.470326070024186 Episode: 197: Time: 19.414796829223633 Total Reward: 292.32394366197076 Avg_Loss: 5.850864065294506 Episode: 198: Time: 19.645657300949097 Total Reward: 413.1967213114725 Avg_Loss: 6.403140197281076 Episode: 199: Time: 19.459718942642212 Total Reward: 264.4306049821979 Avg_Loss: 5.647043366892999 Validation Mean Reward: 535.0900398000277 Validation Std Reward: 239.90639985542626 Test Mean Reward: 577.6411840331049 Test Std Reward: 143.49586544697996
([0.820937740907168, 0.8298551891969532, 0.7261427032346485, 0.678266048055737, 0.6597021974451026, 0.6943003801988954, 0.7574385171051786, 0.6477190712858149, 0.641771079080195, 0.6105592137656543, 0.6308907887765339, 0.6612010868031437, 0.7037577274247628, 0.7539445211437821, 0.784094048930066, 0.7451601366849006, 0.7497997194716409, 0.862367572028096, 0.9059205599943129, 1.0743605771861156, 1.2333044959466999, 1.3613839353896489, 1.299452933130645, 1.1972777161966353, 1.3706221135348833, 1.5413459511865086, 1.854454691670522, 1.9144588631241262, 2.0782367310353687, 2.033498555672269, 2.2022186645439694, 2.584135141442804, 2.7975101751439713, 3.13859794044695, 2.9012930012550675, 2.7518174052238464, 2.602452131629992, 2.490580059149686, 2.676145635232204, 3.0447231716468552, 3.0021196769065215, 3.218664873422695, 3.4549734439669537, 3.540909213178298, 3.669427632784643, 3.413881740650209, 3.255634562308047, 3.78113797682674, 3.7284614934640774, 4.425017001749087, 4.301955237608998, 4.302668186546374, 4.197580825380919, 4.513498431494256, 4.808122724044223, 4.391635022243531, 4.361524848627443, 4.647138198383716, 4.402270266488821, 4.364726959657268, 4.116379081201153, 4.242655820205432, 4.650930493819613, 4.646883855847752, 4.924998957569859, 4.544403590574986, 4.566294788813391, 4.180092493526074, 4.257498892916351, 4.409875603283153, 4.501948399203164, 4.583714746126608, 4.999942636289516, 4.677865306369397, 4.841272954680338, 4.446507530553, 4.844482975346701, 4.84062044860936, 5.348679126310749, 4.451933546226566, 4.64440794351722, 5.2193897166172, 4.961310871008064, 5.2851335305434, 8.26195216930213, 5.074071414330426, 4.938717095791793, 5.031276339743318, 4.914846381720374, 5.255969357089836, 6.335310681026523, 5.407486358610522, 5.353176939387281, 6.522208352048858, 6.568047005589269, 5.1776827678960915, 5.279811822065787, 5.313009090283337, 4.862757809021893, 4.713422799310765, 4.990601285665977, 4.774985970569258, 5.171170523687571, 5.379169203654057, 5.629865767575112, 5.378657738200757, 5.383543312048712, 5.486103039829671, 5.794330969578078, 5.099011708207491, 5.722746386998842, 6.2354671404141335, 5.5336092680442235, 5.137194256822602, 4.9802546240702394, 5.230214607815783, 5.346893361135691, 5.562489731972959, 5.90870685737674, 5.467991328539968, 5.37709853829456, 5.1312863911901205, 5.593012961018987, 5.43684516584172, 5.4918706767699295, 5.841477891978095, 5.773884406109818, 5.462210106248615, 5.614704013371668, 5.689391327505352, 5.709828709854799, 6.069051140496711, 5.868505199416345, 5.795209387270343, 5.797882557917042, 5.962688826713242, 5.61739564342659, 5.707876933722937, 6.475288034487171, 5.259320781511419, 5.713927576521866, 5.316184337399587, 5.938543642268462, 5.671118972682152, 5.597070894321473, 5.46676346133737, 5.292451065628469, 5.058371685132259, 5.165304880683162, 5.594748966834125, 5.644749094458187, 5.8408784756139545, 5.450479232964396, 4.905775434830609, 5.367668441363743, 5.321975791153788, 5.308318919995251, 5.4092133826568345, 5.396987089088985, 5.0815691832734755, 5.425141758277636, 5.237884370218806, 5.445181433894053, 5.611861747353017, 5.902027753220887, 5.636468718532755, 6.263388074245773, 6.873990818732927, 6.404269650202839, 6.263270647585893, 6.352828298296247, 6.045260823073507, 6.507352567520462, 6.223971242664241, 6.358647546848329, 6.103919041256945, 6.29172532999215, 6.297825663029647, 6.3850349132754225, 6.297533519127789, 5.746362517861759, 5.627700875286295, 5.743352380119452, 5.688690711470211, 5.232670697845331, 5.262674479424453, 5.1694381036678285, 5.222657639439366, 5.378060609853568, 5.652116600204916, 5.619969391021408, 6.401196401660182, 5.665336586848027, 5.662818333681892, 5.443334556427322, 5.470326070024186, 5.850864065294506, 6.403140197281076, 5.647043366892999], [-31.708860759494037, -40.762711864407095, -41.20253164557029, -43.08743169398939, -37.942942942943475, -32.95620437956265, -30.06493506493574, -30.251798561151407, -85.25974025973977, -63.30985915493014, -16.14695340501838, -25.402930402931116, 33.289473684209526, -22.083333333334075, -30.01805054151692, -31.090225563910515, -26.899641577061658, -13.831168831169647, 8.806228373702739, 51.42857142857394, 228.23232323232378, -37.85714285714354, -5.493827160494616, -5.000000000000909, 273.60068259385565, 141.74911660777806, 78.14487632509119, 24.60132890365381, 188.01886792453118, 9.615384615383997, 420.5709342560441, 180.00000000000392, 268.6363636363591, 182.372262773726, 197.09621993127547, 388.6363636363559, 102.32441471572366, 7.3890784982942606, 279.9999999999965, 198.28621908127596, 196.81494661922162, 252.1337579617767, 338.5664335664332, 410.5762081784316, 3.8372093023251868, 94.54248366013526, 225.0000000000045, 343.35616438355765, 148.636363636366, 340.9999999999974, 358.90070921985466, 184.2792792792833, 127.58064516129335, 418.6186770427905, 389.3205574912811, 160.5555555555564, 235.33033033033084, 242.79264214047328, 277.88135593219675, 188.48909657321346, 275.0000000000034, 432.5862068965424, 317.37113402061823, 212.69230769231112, 5.694444444443539, 290.3820598006585, 304.9999999999919, 322.5084175084156, 218.43283582089936, 226.87500000000426, 368.8157894736778, 297.739273927393, 204.5951417004059, 380.2851711026554, 237.2784810126617, 137.4840764331252, 173.96551724138106, 309.5801526717537, 371.8989547038308, 148.33333333333738, 252.9729729729689, 315.95890410958356, 286.41025641025595, -61.44905660377387, 151.79487179487413, 28.55212355212548, 342.7224199288254, 425.5992509363207, 241.50793650793923, 500.6678700360931, 337.5259515570859, 392.3646209386246, 212.97101449275738, 220.2866242038258, 413.30564784052433, 382.8481012658181, 317.96928327645077, 450.1388888888868, 382.35191637630163, 367.897526501758, 326.64179104477273, 342.69968051117945, 453.5074626865611, 397.7007299270019, 93.40579710145292, 456.85185185184423, 322.5824175824099, 66.18421052631804, 368.1578947368381, 353.27586206896274, 300.83333333332536, 375.17543859648663, 327.2972972972949, 381.190476190472, 455.5226480836172, 146.071428571432, 189.21052631579312, 151.00638977636208, 319.2857142857122, 522.4496644295259, 326.56862745097715, 227.0973782771575, 349.06779661016003, 85.88737201365646, 302.00374531834535, 443.7596899224762, 411.451612903221, 436.135531135522, 456.85185185183855, 432.5080906148812, 427.5563909774394, 124.69696969697227, 287.3529411764724, 275.4918032786851, 287.75862068964636, 306.81268882175084, 733.3582089552098, 450.7627118644, 307.73037542661507, 105.65789473684507, 493.85017421601646, 213.82352941176848, 476.99999999999216, 237.21476510067126, 387.6589595375691, 188.43949044586367, 455.3144654087988, 488.0115830115729, 181.96793002915885, 428.02631578946574, 452.2312703583, 426.3114754098273, 482.9467680608253, 304.22480620154926, 240.74007220217027, 488.3333333333272, 401.79487179486443, 134.7297297297336, 196.82879377432312, 546.1960132890242, 497.92035398229496, 328.3576642335696, 436.6901408450667, 335.3405572755354, 394.6551724137886, 453.821548821536, 377.89156626505496, 616.1913357400624, 419.1843971631158, 460.92105263156884, 261.401384083048, 354.43820224718337, 240.54817275747786, 342.29903536977037, 468.8297872340382, 345.97222222221814, 478.4265734265687, 401.71052631578084, 532.9069767441736, 464.84555984555345, 455.1618122977303, 244.6226415094381, 281.0932944606431, 345.0000000000007, 449.802867383509, 434.41176470587476, 519.8148148148039, 42.25490196078759, 423.39464882942525, 516.1111111111027, 469.102564102555, 390.0498338870405, 334.41176470587595, 376.83098591548776, 297.5233644859756, 268.36336336336586, 292.32394366197076, 413.1967213114725, 264.4306049821979], [201.8575741131731, 445.9416159286664, 376.4456516851086, 535.0900398000277], [160.6668853162569, 163.82271763193447, 140.30196664980343, 239.90639985542626])
total_rewards, frames = trainerHardUpdateDQN.play_episode(0,True,42)anim = animate(frames)HTML(anim.to_jshtml())xxxxxxxxxxtraineSoftUpdateDQN = DQN.SoftUpdateDQN(EnvWrapper(env), model.Nature_Paper_Conv, tau = 0.3, update_freq = 1, lr = 0.00025, gamma = 0.95, buffer_size=100000, batch_size=32, loss_fn = "mse_loss", use_wandb = False, device = 'cpu', seed = 42, epsilon_scheduler = utils.exponential_decay(1, 1000,0.1), save_path = utils.get_save_path("DoubleDQN_SoftUpdates","./runs/"))traineSoftUpdateDQN.train(200,50,30,50,50)saving to ./runs/DoubleDQN_SoftUpdates/run1 Episode: 1: Time: 18.07413673400879 Total Reward: -27.659932659933013 Avg_Loss: 0.8300108550840123 Episode: 2: Time: 24.553028106689453 Total Reward: -49.385964912281366 Avg_Loss: 0.7992296721629736 Episode: 3: Time: 24.530145168304443 Total Reward: -25.37974683544328 Avg_Loss: 0.7479331327774444 Episode: 4: Time: 24.336063861846924 Total Reward: -37.37288135593228 Avg_Loss: 0.7622552723193369 Episode: 5: Time: 24.379772901535034 Total Reward: -28.54430379746848 Avg_Loss: 0.7252162662373871 Episode: 6: Time: 24.57474374771118 Total Reward: -29.426229508197164 Avg_Loss: 0.7652347375552694 Episode: 7: Time: 24.163302183151245 Total Reward: -37.94294294294362 Avg_Loss: 0.8004983032137907 Episode: 8: Time: 23.828334093093872 Total Reward: -29.30656934306619 Avg_Loss: 0.735148368418968 Episode: 9: Time: 23.912610054016113 Total Reward: -36.55844155844227 Avg_Loss: 0.7089617055754701 Episode: 10: Time: 23.95942711830139 Total Reward: -33.84892086331017 Avg_Loss: 0.6998507880425754 Episode: 11: Time: 24.020150184631348 Total Reward: -52.792207792208174 Avg_Loss: 0.6667009149091083 Episode: 12: Time: 24.01817274093628 Total Reward: 45.84507042253696 Avg_Loss: 0.7642510254092577 Episode: 13: Time: 23.950135946273804 Total Reward: 26.8637992831529 Avg_Loss: 0.8437968341931075 Episode: 14: Time: 23.955140113830566 Total Reward: -10.75091575091625 Avg_Loss: 1.0328185649103476 Episode: 15: Time: 24.408271074295044 Total Reward: 59.605263157898214 Avg_Loss: 1.0345290218706893 Episode: 16: Time: 24.509673833847046 Total Reward: -11.666666666667028 Avg_Loss: 1.1972487048566842 Episode: 17: Time: 24.109836101531982 Total Reward: 53.01444043321325 Avg_Loss: 1.4567115618025555 Episode: 18: Time: 23.80960512161255 Total Reward: 44.09774436090474 Avg_Loss: 1.5350821574195093 Episode: 19: Time: 23.956830978393555 Total Reward: -55.573476702509694 Avg_Loss: 1.4080379387410749 Episode: 20: Time: 24.08735418319702 Total Reward: 125.77922077922494 Avg_Loss: 1.5451214755533123 Episode: 21: Time: 23.88269305229187 Total Reward: 91.85121107266843 Avg_Loss: 1.530254575572595 Episode: 22: Time: 23.7515869140625 Total Reward: 51.42857142857505 Avg_Loss: 1.885333669411034 Episode: 23: Time: 23.984251022338867 Total Reward: 309.040404040392 Avg_Loss: 2.2900999418827666 Episode: 24: Time: 24.015458822250366 Total Reward: 51.03174603174819 Avg_Loss: 2.5265566681613443 Episode: 25: Time: 23.87544083595276 Total Reward: 99.44444444444883 Avg_Loss: 2.756238182051843 Episode: 26: Time: 23.934540033340454 Total Reward: 128.33333333333653 Avg_Loss: 2.502442905381948 Episode: 27: Time: 23.906362056732178 Total Reward: 31.279863481230414 Avg_Loss: 2.1546780967912755 Episode: 28: Time: 24.163328886032104 Total Reward: 364.3639575971679 Avg_Loss: 2.59941013733379 Episode: 29: Time: 23.847429275512695 Total Reward: 35.74204946996592 Avg_Loss: 3.0896950704710826 Episode: 30: Time: 24.171907901763916 Total Reward: 210.64784053156268 Avg_Loss: 2.699759793381731 Episode: 31: Time: 23.68150782585144 Total Reward: 146.50943396226717 Avg_Loss: 3.049805957980517 Episode: 32: Time: 24.21736192703247 Total Reward: 101.92307692308042 Avg_Loss: 3.4621525126345016 Episode: 33: Time: 23.896690845489502 Total Reward: 199.11764705882737 Avg_Loss: 3.1284506323457766 Episode: 34: Time: 24.22518014907837 Total Reward: 170.62500000000412 Avg_Loss: 3.183662277059395 Episode: 35: Time: 23.944808959960938 Total Reward: 359.5454545454502 Avg_Loss: 3.6082325252164313 Episode: 36: Time: 23.837024927139282 Total Reward: 288.2116788321096 Avg_Loss: 3.8849641199873277 Episode: 37: Time: 23.94980216026306 Total Reward: 128.36769759450567 Avg_Loss: 3.9139280279143516 Episode: 38: Time: 23.672550916671753 Total Reward: 294.0909090908989 Avg_Loss: 3.6640041653348616 Episode: 39: Time: 24.003576040267944 Total Reward: 560.5183946488212 Avg_Loss: 3.645185566249014 Episode: 40: Time: 23.86524796485901 Total Reward: 369.1638225255909 Avg_Loss: 3.7821127163762807 Episode: 41: Time: 24.033586978912354 Total Reward: 395.131578947357 Avg_Loss: 4.0907911127354915 Episode: 42: Time: 23.93858528137207 Total Reward: 502.17314487631836 Avg_Loss: 4.557052588763357 Episode: 43: Time: 23.893703937530518 Total Reward: 463.71886120995896 Avg_Loss: 4.904789734287422 Episode: 44: Time: 23.946732759475708 Total Reward: 357.22929936305064 Avg_Loss: 5.569140385679838 Episode: 45: Time: 23.924391984939575 Total Reward: 265.1398601398608 Avg_Loss: 5.347582130872903 Episode: 46: Time: 24.066622972488403 Total Reward: 339.9442379182123 Avg_Loss: 5.541219190890048 Episode: 47: Time: 24.20013117790222 Total Reward: 137.55813953488396 Avg_Loss: 4.510031675090309 Episode: 48: Time: 24.12608313560486 Total Reward: 91.27450980392305 Avg_Loss: 5.326200010395851 Episode: 49: Time: 24.13723397254944 Total Reward: 361.66666666666197 Avg_Loss: 6.18831264772335 Validation Mean Reward: 471.45867341524246 Validation Std Reward: 160.79906606839026 Episode: 50: Time: 24.69616985321045 Total Reward: 696.8215613382775 Avg_Loss: 5.690089408089133 Episode: 51: Time: 25.041243076324463 Total Reward: 113.86075949367115 Avg_Loss: 5.880393457012016 Episode: 52: Time: 24.947258234024048 Total Reward: 277.00000000000375 Avg_Loss: 5.847527385258875 Episode: 53: Time: 24.433032989501953 Total Reward: 252.51773049645692 Avg_Loss: 5.743589263002412 Episode: 54: Time: 24.528048276901245 Total Reward: 181.27627627627797 Avg_Loss: 6.131293151558948 Episode: 55: Time: 24.377076148986816 Total Reward: 111.45161290322699 Avg_Loss: 5.996091170972135 Episode: 56: Time: 24.28488779067993 Total Reward: 469.2023346303453 Avg_Loss: 6.451809533002998 Episode: 57: Time: 24.082210063934326 Total Reward: 382.3519163762976 Avg_Loss: 6.098300312747474 Episode: 58: Time: 24.321101665496826 Total Reward: 210.55555555555793 Avg_Loss: 5.751846659083326 Episode: 59: Time: 24.27101492881775 Total Reward: 190.28528528528926 Avg_Loss: 6.2616657970332295 Episode: 60: Time: 23.964522123336792 Total Reward: 232.75919732441963 Avg_Loss: 6.354010843429245 Episode: 61: Time: 24.034055948257446 Total Reward: 328.72881355931537 Avg_Loss: 6.382988461927206 Episode: 62: Time: 23.855509996414185 Total Reward: 225.87227414330624 Avg_Loss: 5.672853859532781 Episode: 63: Time: 23.79921531677246 Total Reward: 484.99999999998937 Avg_Loss: 5.615620982747118 Episode: 64: Time: 23.832756996154785 Total Reward: 353.27586206896297 Avg_Loss: 6.4947488898990535 Episode: 65: Time: 23.946813821792603 Total Reward: 403.28178694157555 Avg_Loss: 5.964294207196276 Episode: 66: Time: 24.087135076522827 Total Reward: 404.99999999999534 Avg_Loss: 5.775175348550332 Episode: 67: Time: 23.9680597782135 Total Reward: 467.4999999999918 Avg_Loss: 5.796006438611936 Episode: 68: Time: 24.047780990600586 Total Reward: 429.9169435215853 Avg_Loss: 6.218490439302781 Episode: 69: Time: 23.953991174697876 Total Reward: 378.684210526313 Avg_Loss: 6.010284955762014 Episode: 70: Time: 23.89307403564453 Total Reward: 238.33333333333783 Avg_Loss: 6.60006219899955 Episode: 71: Time: 24.063868045806885 Total Reward: 224.40298507463027 Avg_Loss: 6.9699161423354585 Episode: 72: Time: 23.96518087387085 Total Reward: 308.12499999999915 Avg_Loss: 6.235188206203845 Episode: 73: Time: 23.818681955337524 Total Reward: 227.36842105262778 Avg_Loss: 6.011000535568269 Episode: 74: Time: 23.87460994720459 Total Reward: 363.74587458745134 Avg_Loss: 5.679340079051106 Episode: 75: Time: 23.69860601425171 Total Reward: 431.31578947367944 Avg_Loss: 5.503821203688614 Episode: 76: Time: 23.704572200775146 Total Reward: 384.0874524714807 Avg_Loss: 6.564495843498647 Episode: 77: Time: 23.843311071395874 Total Reward: 230.9493670886118 Avg_Loss: 6.434193434835482 Episode: 78: Time: 23.971745014190674 Total Reward: 347.6751592356607 Avg_Loss: 6.4383740936006815 Episode: 79: Time: 23.77141499519348 Total Reward: 377.41379310344496 Avg_Loss: 6.007536951233359 Episode: 80: Time: 23.667709827423096 Total Reward: 229.42748091603522 Avg_Loss: 6.562487988912759 Episode: 81: Time: 23.66022801399231 Total Reward: 452.03832752612186 Avg_Loss: 5.683545070535996 Episode: 82: Time: 23.765236854553223 Total Reward: 418.3333333333264 Avg_Loss: 6.575726713453021
Episode: 83: Time: 23.76634192466736 Total Reward: 222.56756756757227 Avg_Loss: 6.366574479752228 Episode: 84: Time: 23.758791208267212 Total Reward: 538.5616438356057 Avg_Loss: 7.061873446993467 Episode: 85: Time: 23.785446882247925 Total Reward: 344.10256410255903 Avg_Loss: 6.663345629427614 Episode: 86: Time: 23.59667682647705 Total Reward: 455.9433962264061 Avg_Loss: 6.096195738355653 Episode: 87: Time: 23.800740242004395 Total Reward: 501.15384615384147 Avg_Loss: 6.181015587654434 Episode: 88: Time: 23.62913990020752 Total Reward: 372.18146718146676 Avg_Loss: 6.287966750249141 Episode: 89: Time: 23.59816026687622 Total Reward: 513.5409252668956 Avg_Loss: 6.040209119059458 Episode: 90: Time: 23.558493852615356 Total Reward: 639.0823970037354 Avg_Loss: 6.148835222260291 Episode: 91: Time: 23.786091089248657 Total Reward: 381.1904761904707 Avg_Loss: 6.140472198233885 Episode: 92: Time: 23.58467197418213 Total Reward: 414.0252707581153 Avg_Loss: 6.393856681695505 Episode: 93: Time: 23.672492027282715 Total Reward: 444.79238754324894 Avg_Loss: 6.821055284568241 Episode: 94: Time: 23.644577980041504 Total Reward: 771.4259927797734 Avg_Loss: 6.131326370880384 Episode: 95: Time: 23.643855810165405 Total Reward: 390.50724637680844 Avg_Loss: 6.311766629459477 Episode: 96: Time: 23.79970097541809 Total Reward: 404.99999999999346 Avg_Loss: 6.40082303716355 Episode: 97: Time: 23.67268991470337 Total Reward: 396.69435215946316 Avg_Loss: 7.073876817687219 Episode: 98: Time: 23.69080114364624 Total Reward: 136.0126582278525 Avg_Loss: 6.958047385977096 Episode: 99: Time: 23.674755096435547 Total Reward: 372.5767918088723 Avg_Loss: 6.504879835273037 Validation Mean Reward: 559.5188354347463 Validation Std Reward: 185.0716164594511 Episode: 100: Time: 23.773072004318237 Total Reward: 383.5478547854775 Avg_Loss: 6.325964882594197 Episode: 101: Time: 23.62119770050049 Total Reward: 354.6402877697776 Avg_Loss: 6.3860986618434685 Episode: 102: Time: 23.62396812438965 Total Reward: 378.49823321554095 Avg_Loss: 6.448535365216872 Episode: 103: Time: 23.656458854675293 Total Reward: 550.5223880596914 Avg_Loss: 6.885855807977564 Episode: 104: Time: 23.71588683128357 Total Reward: 262.8274760383418 Avg_Loss: 6.844434656515843 Episode: 105: Time: 23.67239499092102 Total Reward: 371.41791044775465 Avg_Loss: 6.5723879237134915 Episode: 106: Time: 23.57415223121643 Total Reward: 445.14598540145414 Avg_Loss: 6.370161876958959 Episode: 107: Time: 23.616703033447266 Total Reward: 419.49275362318167 Avg_Loss: 6.318225161868985 Episode: 108: Time: 23.582480907440186 Total Reward: 442.0370370370274 Avg_Loss: 6.9709720010517024 Episode: 109: Time: 23.64343285560608 Total Reward: 582.655677655669 Avg_Loss: 6.8816664499395035 Episode: 110: Time: 23.745257139205933 Total Reward: 197.76315789474097 Avg_Loss: 7.341861600635433 Episode: 111: Time: 23.60998296737671 Total Reward: 473.42105263157174 Avg_Loss: 6.729159209908557 Episode: 112: Time: 23.785827159881592 Total Reward: 394.6551724137886 Avg_Loss: 7.045944719755349 Episode: 113: Time: 23.816050052642822 Total Reward: 415.4166666666619 Avg_Loss: 7.036238842651624 Episode: 114: Time: 23.707013845443726 Total Reward: 455.8771929824469 Avg_Loss: 6.730546558604521 Episode: 115: Time: 23.816734313964844 Total Reward: 337.43243243242875 Avg_Loss: 6.824104431296597 Episode: 116: Time: 23.57311511039734 Total Reward: 443.46153846153254 Avg_Loss: 6.972022191817019 Episode: 117: Time: 23.62332320213318 Total Reward: 326.60278745644655 Avg_Loss: 6.686858927502351 Episode: 118: Time: 23.832597017288208 Total Reward: 315.714285714284 Avg_Loss: 6.570290589532933 Episode: 119: Time: 23.645842790603638 Total Reward: 129.56140350877632 Avg_Loss: 6.409988800016772 Episode: 120: Time: 23.867870092391968 Total Reward: 317.1405750798675 Avg_Loss: 6.763044659830943 Episode: 121: Time: 23.564437866210938 Total Reward: 197.8571428571479 Avg_Loss: 6.8811724967315415 Episode: 122: Time: 23.682533979415894 Total Reward: 294.2617449664447 Avg_Loss: 7.0000414217219635 Episode: 123: Time: 23.77399182319641 Total Reward: 382.12418300652706 Avg_Loss: 6.437855581275556 Episode: 124: Time: 23.621232271194458 Total Reward: 339.4569288389487 Avg_Loss: 6.7764776253900605 Episode: 125: Time: 23.74714493751526 Total Reward: 586.3559322033791 Avg_Loss: 6.90538028709027 Episode: 126: Time: 23.800681114196777 Total Reward: 420.3583617747355 Avg_Loss: 6.964396986640802 Episode: 127: Time: 23.615633010864258 Total Reward: 552.9400749063561 Avg_Loss: 6.780861733340416 Episode: 128: Time: 23.594789028167725 Total Reward: 618.1782945736355 Avg_Loss: 6.787586170084336 Episode: 129: Time: 23.719347953796387 Total Reward: 485.6451612903146 Avg_Loss: 6.557832578651044 Episode: 130: Time: 23.71891188621521 Total Reward: 674.230769230758 Avg_Loss: 6.915215129611873 Episode: 131: Time: 23.55812406539917 Total Reward: 345.74074074073667 Avg_Loss: 7.156246171278112 Episode: 132: Time: 23.75821304321289 Total Reward: 516.6504854368816 Avg_Loss: 7.641616690058668 Episode: 133: Time: 23.60245108604431 Total Reward: 476.4285714285617 Avg_Loss: 7.108696390600765 Episode: 134: Time: 23.62402629852295 Total Reward: 567.8787878787812 Avg_Loss: 6.978704779087996 Episode: 135: Time: 23.752808094024658 Total Reward: 382.1241830065313 Avg_Loss: 7.081404710016331 Episode: 136: Time: 23.87787890434265 Total Reward: 485.3278688524497 Avg_Loss: 6.930347182169682 Episode: 137: Time: 23.783859968185425 Total Reward: 491.20689655171304 Avg_Loss: 6.6143887443702765 Episode: 138: Time: 23.877455949783325 Total Reward: 352.1299093655544 Avg_Loss: 7.192500962930567 Episode: 139: Time: 23.54002094268799 Total Reward: 382.6119402985041 Avg_Loss: 6.844065774388674 Episode: 140: Time: 23.812363147735596 Total Reward: 403.30508474575583 Avg_Loss: 6.800672771550026 Episode: 141: Time: 23.838149070739746 Total Reward: 567.116040955618 Avg_Loss: 6.417030947549002 Episode: 142: Time: 23.751939296722412 Total Reward: 352.3684210526216 Avg_Loss: 6.70686987067471 Episode: 143: Time: 23.71762990951538 Total Reward: 563.5365853658438 Avg_Loss: 6.619678638562434 Episode: 144: Time: 23.960188150405884 Total Reward: 487.35294117646276 Avg_Loss: 6.466956207732193 Episode: 145: Time: 23.64506506919861 Total Reward: 560.9999999999893 Avg_Loss: 6.513240405491421 Episode: 146: Time: 23.737879037857056 Total Reward: 381.51006711408917 Avg_Loss: 7.393151524688015 Episode: 147: Time: 23.988882064819336 Total Reward: 257.6011560693589 Avg_Loss: 6.926842789690034 Episode: 148: Time: 23.793062210083008 Total Reward: 306.2738853503195 Avg_Loss: 7.000780734695306 Episode: 149: Time: 23.791245222091675 Total Reward: 99.96855345912417 Avg_Loss: 7.117604085377285 Validation Mean Reward: 652.840052991201 Validation Std Reward: 183.57966331963064 Episode: 150: Time: 23.691931009292603 Total Reward: 598.1034482758492 Avg_Loss: 6.610824140180059 Episode: 151: Time: 23.815236806869507 Total Reward: 406.68350168349514 Avg_Loss: 6.658065924624435 Episode: 152: Time: 23.83064079284668 Total Reward: 421.4473684210418 Avg_Loss: 6.888720248927589 Episode: 153: Time: 23.867711067199707 Total Reward: 344.7394136807789 Avg_Loss: 6.838023745713114 Episode: 154: Time: 23.896119117736816 Total Reward: 475.4918032786821 Avg_Loss: 7.141655969018696 Episode: 155: Time: 23.68617820739746 Total Reward: 494.3536121672961 Avg_Loss: 6.517667807450815 Episode: 156: Time: 23.896977186203003 Total Reward: 614.3023255813905 Avg_Loss: 6.665090915535679 Episode: 157: Time: 23.73231291770935 Total Reward: 468.17689530685277 Avg_Loss: 6.828455363001142 Episode: 158: Time: 23.656770944595337 Total Reward: 443.19444444443997 Avg_Loss: 7.356824612417141 Episode: 159: Time: 23.786828994750977 Total Reward: 430.6410256410195 Avg_Loss: 6.530079206498731 Episode: 160: Time: 23.49199104309082 Total Reward: 549.144144144129 Avg_Loss: 7.098329968813087 Episode: 161: Time: 23.643954277038574 Total Reward: 422.50972762644744 Avg_Loss: 6.155594136534619 Episode: 162: Time: 23.748470067977905 Total Reward: 536.2292358803891 Avg_Loss: 6.809823951801332 Episode: 163: Time: 23.94558596611023 Total Reward: 379.9262536873142 Avg_Loss: 6.848604566910687 Episode: 164: Time: 23.68020272254944 Total Reward: 576.5328467153171 Avg_Loss: 6.9952815530680805
Episode: 165: Time: 23.628339767456055 Total Reward: 570.4929577464707 Avg_Loss: 7.286853023937771 Episode: 166: Time: 23.989874124526978 Total Reward: 511.8111455108276 Avg_Loss: 6.479278967160137 Episode: 167: Time: 23.811520099639893 Total Reward: 625.6896551724005 Avg_Loss: 6.699736808528419 Episode: 168: Time: 23.83064603805542 Total Reward: 460.55555555555134 Avg_Loss: 6.457543641579251 Episode: 169: Time: 23.96686625480652 Total Reward: 447.1686746987889 Avg_Loss: 7.201328754425049 Episode: 170: Time: 23.71521782875061 Total Reward: 479.00722021659595 Avg_Loss: 7.818225259540462 Episode: 171: Time: 23.818684816360474 Total Reward: 326.98581560283804 Avg_Loss: 7.183639678634515 Episode: 172: Time: 23.79445791244507 Total Reward: 319.4736842105234 Avg_Loss: 7.154375471988646 Episode: 173: Time: 23.740927934646606 Total Reward: 558.9792387543155 Avg_Loss: 6.946575574514244 Episode: 174: Time: 23.425029039382935 Total Reward: 32.340823970040006 Avg_Loss: 6.7369162886082625 Episode: 175: Time: 23.790561199188232 Total Reward: 373.4385382059759 Avg_Loss: 6.6599887774772 Episode: 176: Time: 23.936282873153687 Total Reward: 499.85530546622977 Avg_Loss: 6.908925186686155 Episode: 177: Time: 23.788964986801147 Total Reward: 529.113475177295 Avg_Loss: 6.486106440800579 Episode: 178: Time: 23.885277032852173 Total Reward: 214.02777777778263 Avg_Loss: 7.535218429164726 Episode: 179: Time: 23.821508169174194 Total Reward: 464.4405594405536 Avg_Loss: 6.935675879486468 Episode: 180: Time: 23.906039237976074 Total Reward: 460.92105263157066 Avg_Loss: 7.469788257815257 Episode: 181: Time: 23.86699914932251 Total Reward: 243.87043189368396 Avg_Loss: 7.4392158123625425 Episode: 182: Time: 23.65690279006958 Total Reward: 711.9498069497934 Avg_Loss: 6.781666351967499 Episode: 183: Time: 23.987497329711914 Total Reward: 442.2168284789582 Avg_Loss: 7.120240560098856 Episode: 184: Time: 24.015455961227417 Total Reward: 436.4465408804971 Avg_Loss: 6.803372717705093 Episode: 185: Time: 24.13535785675049 Total Reward: 391.8804664723 Avg_Loss: 7.476550602111496 Episode: 186: Time: 23.790990114212036 Total Reward: 370.4545454545424 Avg_Loss: 7.000144192150661 Episode: 187: Time: 23.732338190078735 Total Reward: 489.22939068099316 Avg_Loss: 7.7867571265757585 Episode: 188: Time: 23.865803003311157 Total Reward: 663.1699346405114 Avg_Loss: 7.15203496888906 Episode: 189: Time: 23.775097846984863 Total Reward: 716.1111111110968 Avg_Loss: 7.0840509689154745 Episode: 190: Time: 23.646601915359497 Total Reward: 587.3529411764597 Avg_Loss: 6.996306042711274 Episode: 191: Time: 23.956536054611206 Total Reward: 259.51505016722814 Avg_Loss: 7.924688359268573 Episode: 192: Time: 23.852011919021606 Total Reward: 477.9166666666608 Avg_Loss: 7.335046261298556 Episode: 193: Time: 23.750523805618286 Total Reward: 414.1575091574999 Avg_Loss: 6.85980285616482 Episode: 194: Time: 23.85342788696289 Total Reward: 516.2956810631151 Avg_Loss: 7.34422272694211 Episode: 195: Time: 24.067399263381958 Total Reward: 219.7058823529453 Avg_Loss: 7.134669357989015 Episode: 196: Time: 23.833743810653687 Total Reward: 426.126760563374 Avg_Loss: 7.611266607997798 Episode: 197: Time: 23.847986936569214 Total Reward: 409.6728971962542 Avg_Loss: 7.097862505111374 Episode: 198: Time: 24.005181074142456 Total Reward: 322.4174174174168 Avg_Loss: 7.011147813636716 Episode: 199: Time: 23.880630016326904 Total Reward: 552.8873239436507 Avg_Loss: 7.443516914584055 Validation Mean Reward: 638.3100554790802 Validation Std Reward: 186.44568541943735 Test Mean Reward: 703.9483103332115 Test Std Reward: 130.17643986814338
([0.8300108550840123, 0.7992296721629736, 0.7479331327774444, 0.7622552723193369, 0.7252162662373871, 0.7652347375552694, 0.8004983032137907, 0.735148368418968, 0.7089617055754701, 0.6998507880425754, 0.6667009149091083, 0.7642510254092577, 0.8437968341931075, 1.0328185649103476, 1.0345290218706893, 1.1972487048566842, 1.4567115618025555, 1.5350821574195093, 1.4080379387410749, 1.5451214755533123, 1.530254575572595, 1.885333669411034, 2.2900999418827666, 2.5265566681613443, 2.756238182051843, 2.502442905381948, 2.1546780967912755, 2.59941013733379, 3.0896950704710826, 2.699759793381731, 3.049805957980517, 3.4621525126345016, 3.1284506323457766, 3.183662277059395, 3.6082325252164313, 3.8849641199873277, 3.9139280279143516, 3.6640041653348616, 3.645185566249014, 3.7821127163762807, 4.0907911127354915, 4.557052588763357, 4.904789734287422, 5.569140385679838, 5.347582130872903, 5.541219190890048, 4.510031675090309, 5.326200010395851, 6.18831264772335, 5.690089408089133, 5.880393457012016, 5.847527385258875, 5.743589263002412, 6.131293151558948, 5.996091170972135, 6.451809533002998, 6.098300312747474, 5.751846659083326, 6.2616657970332295, 6.354010843429245, 6.382988461927206, 5.672853859532781, 5.615620982747118, 6.4947488898990535, 5.964294207196276, 5.775175348550332, 5.796006438611936, 6.218490439302781, 6.010284955762014, 6.60006219899955, 6.9699161423354585, 6.235188206203845, 6.011000535568269, 5.679340079051106, 5.503821203688614, 6.564495843498647, 6.434193434835482, 6.4383740936006815, 6.007536951233359, 6.562487988912759, 5.683545070535996, 6.575726713453021, 6.366574479752228, 7.061873446993467, 6.663345629427614, 6.096195738355653, 6.181015587654434, 6.287966750249141, 6.040209119059458, 6.148835222260291, 6.140472198233885, 6.393856681695505, 6.821055284568241, 6.131326370880384, 6.311766629459477, 6.40082303716355, 7.073876817687219, 6.958047385977096, 6.504879835273037, 6.325964882594197, 6.3860986618434685, 6.448535365216872, 6.885855807977564, 6.844434656515843, 6.5723879237134915, 6.370161876958959, 6.318225161868985, 6.9709720010517024, 6.8816664499395035, 7.341861600635433, 6.729159209908557, 7.045944719755349, 7.036238842651624, 6.730546558604521, 6.824104431296597, 6.972022191817019, 6.686858927502351, 6.570290589532933, 6.409988800016772, 6.763044659830943, 6.8811724967315415, 7.0000414217219635, 6.437855581275556, 6.7764776253900605, 6.90538028709027, 6.964396986640802, 6.780861733340416, 6.787586170084336, 6.557832578651044, 6.915215129611873, 7.156246171278112, 7.641616690058668, 7.108696390600765, 6.978704779087996, 7.081404710016331, 6.930347182169682, 6.6143887443702765, 7.192500962930567, 6.844065774388674, 6.800672771550026, 6.417030947549002, 6.70686987067471, 6.619678638562434, 6.466956207732193, 6.513240405491421, 7.393151524688015, 6.926842789690034, 7.000780734695306, 7.117604085377285, 6.610824140180059, 6.658065924624435, 6.888720248927589, 6.838023745713114, 7.141655969018696, 6.517667807450815, 6.665090915535679, 6.828455363001142, 7.356824612417141, 6.530079206498731, 7.098329968813087, 6.155594136534619, 6.809823951801332, 6.848604566910687, 6.9952815530680805, 7.286853023937771, 6.479278967160137, 6.699736808528419, 6.457543641579251, 7.201328754425049, 7.818225259540462, 7.183639678634515, 7.154375471988646, 6.946575574514244, 6.7369162886082625, 6.6599887774772, 6.908925186686155, 6.486106440800579, 7.535218429164726, 6.935675879486468, 7.469788257815257, 7.4392158123625425, 6.781666351967499, 7.120240560098856, 6.803372717705093, 7.476550602111496, 7.000144192150661, 7.7867571265757585, 7.15203496888906, 7.0840509689154745, 6.996306042711274, 7.924688359268573, 7.335046261298556, 6.85980285616482, 7.34422272694211, 7.134669357989015, 7.611266607997798, 7.097862505111374, 7.011147813636716, 7.443516914584055], [-27.659932659933013, -49.385964912281366, -25.37974683544328, -37.37288135593228, -28.54430379746848, -29.426229508197164, -37.94294294294362, -29.30656934306619, -36.55844155844227, -33.84892086331017, -52.792207792208174, 45.84507042253696, 26.8637992831529, -10.75091575091625, 59.605263157898214, -11.666666666667028, 53.01444043321325, 44.09774436090474, -55.573476702509694, 125.77922077922494, 91.85121107266843, 51.42857142857505, 309.040404040392, 51.03174603174819, 99.44444444444883, 128.33333333333653, 31.279863481230414, 364.3639575971679, 35.74204946996592, 210.64784053156268, 146.50943396226717, 101.92307692308042, 199.11764705882737, 170.62500000000412, 359.5454545454502, 288.2116788321096, 128.36769759450567, 294.0909090908989, 560.5183946488212, 369.1638225255909, 395.131578947357, 502.17314487631836, 463.71886120995896, 357.22929936305064, 265.1398601398608, 339.9442379182123, 137.55813953488396, 91.27450980392305, 361.66666666666197, 696.8215613382775, 113.86075949367115, 277.00000000000375, 252.51773049645692, 181.27627627627797, 111.45161290322699, 469.2023346303453, 382.3519163762976, 210.55555555555793, 190.28528528528926, 232.75919732441963, 328.72881355931537, 225.87227414330624, 484.99999999998937, 353.27586206896297, 403.28178694157555, 404.99999999999534, 467.4999999999918, 429.9169435215853, 378.684210526313, 238.33333333333783, 224.40298507463027, 308.12499999999915, 227.36842105262778, 363.74587458745134, 431.31578947367944, 384.0874524714807, 230.9493670886118, 347.6751592356607, 377.41379310344496, 229.42748091603522, 452.03832752612186, 418.3333333333264, 222.56756756757227, 538.5616438356057, 344.10256410255903, 455.9433962264061, 501.15384615384147, 372.18146718146676, 513.5409252668956, 639.0823970037354, 381.1904761904707, 414.0252707581153, 444.79238754324894, 771.4259927797734, 390.50724637680844, 404.99999999999346, 396.69435215946316, 136.0126582278525, 372.5767918088723, 383.5478547854775, 354.6402877697776, 378.49823321554095, 550.5223880596914, 262.8274760383418, 371.41791044775465, 445.14598540145414, 419.49275362318167, 442.0370370370274, 582.655677655669, 197.76315789474097, 473.42105263157174, 394.6551724137886, 415.4166666666619, 455.8771929824469, 337.43243243242875, 443.46153846153254, 326.60278745644655, 315.714285714284, 129.56140350877632, 317.1405750798675, 197.8571428571479, 294.2617449664447, 382.12418300652706, 339.4569288389487, 586.3559322033791, 420.3583617747355, 552.9400749063561, 618.1782945736355, 485.6451612903146, 674.230769230758, 345.74074074073667, 516.6504854368816, 476.4285714285617, 567.8787878787812, 382.1241830065313, 485.3278688524497, 491.20689655171304, 352.1299093655544, 382.6119402985041, 403.30508474575583, 567.116040955618, 352.3684210526216, 563.5365853658438, 487.35294117646276, 560.9999999999893, 381.51006711408917, 257.6011560693589, 306.2738853503195, 99.96855345912417, 598.1034482758492, 406.68350168349514, 421.4473684210418, 344.7394136807789, 475.4918032786821, 494.3536121672961, 614.3023255813905, 468.17689530685277, 443.19444444443997, 430.6410256410195, 549.144144144129, 422.50972762644744, 536.2292358803891, 379.9262536873142, 576.5328467153171, 570.4929577464707, 511.8111455108276, 625.6896551724005, 460.55555555555134, 447.1686746987889, 479.00722021659595, 326.98581560283804, 319.4736842105234, 558.9792387543155, 32.340823970040006, 373.4385382059759, 499.85530546622977, 529.113475177295, 214.02777777778263, 464.4405594405536, 460.92105263157066, 243.87043189368396, 711.9498069497934, 442.2168284789582, 436.4465408804971, 391.8804664723, 370.4545454545424, 489.22939068099316, 663.1699346405114, 716.1111111110968, 587.3529411764597, 259.51505016722814, 477.9166666666608, 414.1575091574999, 516.2956810631151, 219.7058823529453, 426.126760563374, 409.6728971962542, 322.4174174174168, 552.8873239436507], [471.45867341524246, 559.5188354347463, 652.840052991201, 638.3100554790802], [160.79906606839026, 185.0716164594511, 183.57966331963064, 186.44568541943735])
total_rewards, frames = traineSoftUpdateDQN.play_episode(0,True,42)anim = animate(frames)HTML(anim.to_jshtml())